redis 集群介绍及docker实现

1 Redis Cluster 介绍

在哨兵 sentinel 机制中，可以解决 redis 高可用问题，即当 master 故障后可以自动将 slave 提升为 master，从而可以保证 redis 服务的正常使用，但是无法解决 redis 单机写入的瓶颈问题，即单机 redis 写入性能受限于单机的内存大小、并发数量、网卡速率等因素。
为了解决单机性能的瓶颈，提高 Redis 性能，可以使用分布式集群的解决方案

早期Redis 分布式集群部署方案:

客户端分区:由客户端程序决定key写分配和写入的redis node，但是需要客户端自己处理写入分配、高可用管理和故障转移等
代理方案:基于三方软件实现 redis proxy，客户端先连接之代理层，由代理层实现 key 的写入分配，对客户端来说是有比较简单，但是对于集群管节点增减相对比较麻烦，而且代理本身也是单点和性能瓶颈。

redis 3.0 版本之后推出了无中心架构的 redis cluster 机制，在无中心的redis集群当中，其每个节点保存当前节点数据和整个集群状态,每个节点都和其他所有节点连接

1.1 什么是Redis Cluster

Redis Cluster（3.0之后引入的）为 Redis 官方提供的一种分布式集群解决方案。它支持在线节点增加和减少。集群中的节点角色可能是主，也可能是从，但需要保证每个主节点都要有对应的从节点，这样保证了其高可用。

Redis Cluster采用了分布式系统的分片（分区）的思路，每个主节点为一个分片，这样也就意味着存储的数据是分散在所有分片中的。当增加节点或删除主节点时，原存储在某个主节点中的数据会自动再次分配到其他主节点。

各节点间是相互通信的，通信端口为各节点Redis服务端口 +10000，这个端口是固定的，所以注意防火墙设置，节点之间通过二进制协议通信，这样的目的是减少带宽消耗。

在 Redis Cluster 中有一个概念 slot（槽位），我们翻译为槽。slot（槽位）数量是固定的，为 16384 个。这些 slot（槽）会均匀地分布到各个节点上。另外 Redis 的键和值会根据 hash 算法存储在对应的 slot（槽位）中。简单讲，对于一个键值对，存的时候在哪里是通过 hash算法算出来的，那么取得时候也会算一下，知道值在哪个 slot 上。根据 slot 编号再到对应的节点上去取。

Redis Cluster无法保证数据的强一致性，这是因为当数据存储时，只要在主节点上存好了，就会告诉客户端存好了，如果等所有从节点上都同步完再跟客户端确认，那么会有很大的延迟，这个对于客户端来讲是无法容忍的。所以，最终 Redis Cluster 只好放弃了数据强一致性，而把性能放在了首位。

分布式数据库，通过分片机制进行数据分布，clustering 内的每个节点仅存数据库的一部分数据；

官方推荐最大的节点数量为1000，至少需要3（Master）+3（Slave）才能建立集群，是无中心的分布式存储架构
，可以在多个节点之间进行数据共享，解决了Redis高可用、可扩展等问题。集群可以将数据自动切分（split）到多个节点，当集群中的某一个节点故障时，redis还可以继续处理客户端的请求。

Codis(豌豆荚) 业界常用的分布式Redis集群解决方案
Redis Cluster(官方)
Cerberus(芒果TV)

1.2 Redis Cluster的特点

高性能

在多分片节点中，将 16384 个槽位，均匀分布到多个分片节点中
存数据时，将 key 做 crc16(key) , 然后和 16384 进行取模，得出槽位值（0-16384之间）
根据计算得出的槽位值，找到相对应的分片节点的主节点，存储到相应槽位上（卡槽作用：决定 key 存放具体的服务器位置，从而实现均摊存放数据）
如果客户端当时连接的节点不是将来要存储的分片节点，分片集群会将客户端连接切换至真正存储节点进行数据存储

高可用

在搭建集群时，会为每一个分片的主节点，对应一个从节点，实现 replicaof 功能，同时当主节点 down，实现类似于 sentinel 的自动 failover 的功能。

Redis Cluster客户端连接任意节点

如图所示，当我们用客户端连接A分片时，如果按照数据的取模，我们想要访问的数据，不在A分片中，那么集群会自动将请求进行转发。

1.3 redis集群数据共享(设计理念)

Redis 集群使用数据分片（sharding）而非一致性哈希（consistency hashing）来实现：一个 Redis 集群包含 16384 个哈希槽（hash slot），数据库中的每个键都属于这 16384 个哈希槽的其中一个，集群使用公式 CRC16(key) % 16384 来计算键 key 属于哪个槽，其中 CRC16(key) 语句用于计算键 key 的 CRC16 校验和

节点 A 负责处理 0 号至 5461 号哈希槽。
节点 B 负责处理 5462 号至 10922 号哈希槽。
节点 C 负责处理 10923 号至 16383 号哈希槽

注意：

一个 Redis 集群包含 16384 个卡槽，可以存放无数个key，一个卡槽可以存放多个不同的key
只有主节点才会分配卡槽，从节点没有卡槽，所以只有主节点可写，从节点只读
Redis 集群不像单机 Redis 那样支持多数据库功能，集群只使用默认的 0 号数据库，并且不能使用 SELECT index 命令。

1.4 Redis Cluster运行机制

所有的 redis 节点彼此互联（PING-PONG机制）,内部使用二进制协议优化传输速度和带宽。

节点的 fail 是通过集群中超过半数的 master 节点检测失效时才生效，即，集群中某个节点的是否失效，是由整个集群中超过半数的节点监测都失效，才能算真正的失效

客户端与 redis 节点直连，不需要中间 proxy 层。客户端不需要连接集群所有节点，连接集群中任何一个可用节点即可，即，客户端不需要proxy即可直接连接redis，应用程序中需要配置有全部的redis服务器IP

redis cluster 把所有的物理节点（redis node）平均映射到 [0-16383] 个槽位（slot）上，cluster 负责维护 node<->slot<->key，读写需要到指定的 redis node 上进行操作，因此有多少个 redis node 相当于 redis 并发扩展了多少倍，每个redis node 承担 16384/N 个槽位

Redis cluster 预先分配 16384 个（slot）槽位，当需要在 redis 集群中写入一个 key-value 的时候，会使用 CRC16(key) mod 16384之后的值，决定将 key 写入值哪一个槽位从而决定写入哪一个 Redis 节点上，从而有效解决单机瓶颈。

1.5 Redis Cluster如何做集群复制

为了使得集群在一部分节点下线或者无法与集群的大多数（majority）节点进行通讯的情况下，仍然可以正常运作， Redis 集群对节点使用了主从复制功能：集群中的每个节点都有 1 个至 N 个复制品（replica），其中一个复制品为主节点（master），而其余的 N-1 个复制品为从节点（slave）。

在之前列举的节点 A 、B 、C 的例子中，如果节点 B 下线了，那么集群将无法正常运行，因为集群找不到节点来处理 5462 号至 10922 号的哈希槽。

假如在创建集群的时候（或者至少在节点 B 下线之前），我们为主节点 B 添加了从节点 B1 ，那么当主节点 B 下线的时候，集群就会将 B1 设置为新的主节点，并让它代替下线的主节点 B ，继续处理 5462 号至 10922 号的哈希槽，这样集群就不会因为主节点 B 的下线而无法正常运作了。

不过如果节点 B 和 B1 都下线的话， Redis 集群还是会停止运作。

集群的复制特性重用了 replicaof 命令的代码，所以集群节点的复制行为和 replicaof 命令的复制行为完全相同。

2 Redis Cluster故障转移

1）在集群里面，节点会对其他节点进行下线检测。

2）当一个主节点下线时，集群里面的其他主节点负责对下线主节点进行故障移。

3）换句话说，集群的节点集成了下线检测和故障转移等类似 Sentinel 的功能。

4）因为 Sentinel 是一个独立运行的监控程序，而集群的下线检测和故障转移等功能是集成在节点里面的，它们的运行模式非常地不同，所以尽管这两者的功能很相似，但集群的实现没有重用 Sentinel 的代码。

2.1 Redis Cluster中执行命令的两种情况

命令发送到了正确的节点：命令要处理的键所在的槽正好是由接收命令的节点负责，那么该节点执行命令，就像单机 Redis 服务器一样。

命令发送到了错误的节点：接收到命令的节点并非处理键所在槽的节点，那么节点将向客户端返回一个转向（redirection）错误，告知客户端应该到哪个节点去执行这个命令，客户端会根据错误提示的信息，重新向正确的节点发送命令。

3 总结

一个 Redis 集群包含 16384 个卡槽，可以存放无数个key，一个卡槽可以存放多个不同的key
只有主节点才会分配卡槽，从节点没有卡槽，所以只有主节点可写，从节点只读，客户端可以连接任何一个主节点进行读写
Redis 集群不像单机 Redis 那样支持多数据库功能，集群只使用默认的 0 号数据库，并且不能使用 SELECT index 命令。
Redis 集群是一个可以在多个 Redis 节点之间进行数据共享的设施（installation）。
Redis集群不支持那些需要同时处理多个键的 Redis 命令（如：mset/mget），因为执行这些命令需要在多个Redis节点之间移动数据，并且在高并发的情况下，这些命令将降低 Redis 集群的性能，并导致不可预测的行为。
支持在线增加、删除节点（即动态扩缩容）
Redis集群通过分区（partition）来提供一定程度的可用性（availability）：即使集群中有一部分节点失效或者无法进行通讯，集群也可以继续处理命令请求。
Redis集群有将数据自动切分（split）到多个节点的能力。

4 Redis Cluster 架构部署

4.1 部署方式介绍

redis cluster 有多种部署方法

原生命令安装
- 理解Redis Cluster架构
- 生产环境不使用
官方工具安装
- 高效、准确
- 生产环境可以使用
自主研发
- 可以实现可视化的自动化部署

4.2 基于Redis 5 的 redis cluster 部署

官方文档：https://redis.io/topics/cluster-tutorial

redis cluster 相关命令

示例：查看 --cluster 选项帮助

[root@redis-master ~]# docker exec -it redis-node /bin/bash -c "redis-cli --cluster help"
Cluster Manager Commands:create         host1:port1 ... hostN:portN--cluster-replicas <arg>check          host:port--cluster-search-multiple-ownersinfo           host:portfix            host:port--cluster-search-multiple-ownersreshard        host:port--cluster-from <arg>--cluster-to <arg>--cluster-slots <arg>--cluster-yes--cluster-timeout <arg>--cluster-pipeline <arg>--cluster-replacerebalance      host:port--cluster-weight <node1=w1...nodeN=wN>--cluster-use-empty-masters--cluster-timeout <arg>--cluster-simulate--cluster-pipeline <arg>--cluster-threshold <arg>--cluster-replaceadd-node       new_host:new_port existing_host:existing_port--cluster-slave--cluster-master-id <arg>del-node       host:port node_idcall           host:port command arg arg .. argset-timeout    host:port millisecondsimport         host:port--cluster-from <arg>--cluster-copy--cluster-replacehelpFor check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.[root@redis-master ~]#

示例：查看 CLUSTER 指令的帮助

[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 CLUSTER HELP"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.1) CLUSTER <subcommand> arg arg ... arg. Subcommands are:2) ADDSLOTS <slot> [slot ...] -- Assign slots to current node.3) BUMPEPOCH -- Advance the cluster config epoch.4) COUNT-failure-reports <node-id> -- Return number of failure reports for <node-id>.5) COUNTKEYSINSLOT <slot> - Return the number of keys in <slot>.6) DELSLOTS <slot> [slot ...] -- Delete slots information from current node.7) FAILOVER [force|takeover] -- Promote current replica node to being a master.8) FORGET <node-id> -- Remove a node from the cluster.9) GETKEYSINSLOT <slot> <count> -- Return key names stored by current node in a slot.
10) FLUSHSLOTS -- Delete current node own slots information.
11) INFO - Return onformation about the cluster.
12) KEYSLOT <key> -- Return the hash slot for <key>.
13) MEET <ip> <port> [bus-port] -- Connect nodes into a working cluster.
14) MYID -- Return the node id.
15) NODES -- Return cluster configuration seen by node. Output format:
16)     <id> <ip:port> <flags> <master> <pings> <pongs> <epoch> <link> <slot> ... <slot>
17) REPLICATE <node-id> -- Configure current node as replica to <node-id>.
18) RESET [hard|soft] -- Reset current node (default: soft).
19) SET-config-epoch <epoch> - Set config epoch of current node.
20) SETSLOT <slot> (importing|migrating|stable|node <node-id>) -- Set slot state.
21) REPLICAS <node-id> -- Return <node-id> replicas.
22) SLOTS -- Return information about slots range mappings. Each range is made of:
23)     start, end, master and replicas IP addresses, ports and ids
[root@redis-master1 ~]#

4.3 创建 redis cluster集群的环境准备

6台服务器，分别是三组master/slave，适用于生产环境
每个redis 节点采用相同的相同的redis版本、相同的密码、硬件配置
所有redis服务器必须没有任何数据
准备六台主机，地址如下

角色	IP	端口	版本	操作系统
Master	172.18.8.17	6379,16379	redis5.0	CentOS 7.9.2009
Master	172.18.8.27	6379,16379	redis5.0	CentOS 7.9.2009
Master	172.18.8.37	6379,16379	redis5.0	CentOS 7.9.2009
Slave	172.18.8.47	6379,16379	redis5.0	CentOS 7.9.2009
Slave	172.18.8.57	6379,16379	redis5.0	CentOS 7.9.2009
Slave	172.18.8.67	6379,16379	redis5.0	CentOS 7.9.2009

另外准备两台用于业务增长的扩容

角色	IP	端口	版本	操作系统
Master	172.18.8.77	6379,16379	redis5.0	CentOS 7.9.2009
Slave	172.18.8.87	6379,16379	redis5.0	CentOS 7.9.2009

4.4 启用 redis cluster 配置

每个节点修改redis配置，必须开启cluster功能的参数

配置（redis.conf）说明

header 1	header 2
bind 0.0.0.0	监听地址
masterauth 123456	建议配置，否则后期的 master 和 slave 主从复制无法成功
requirepass 123456	密码
cluster-enabled yes	取消此行注释,必须开启集群，开启后 redis 进程会有 cluster 标识
cluster-config-file nodes-6379.conf	取消此行注释,此为集群状态文件,记录主从关系及 slot 范围信息, 由 redis cluster 集群自动创建和维护
cluster-require-full-coverage no	默认值为 yes，设为 no 可以防止一个节点不可用导致整个 cluster 不可能

可用命令批量修改

sed -i.bak \
-e 's/bind 127.0.0.1/bind 0.0.0.0/' \
-e '/# masterauth/a masterauth 123456' \
-e '/# requirepass/a requirepass 123456' \
-e '/# cluster-enabled yes/a cluster-enabled yes' \
-e '/# cluster-config-file nodes-6379.conf/a cluster-config-file nodes-6379.conf' \
-e '/cluster-require-full-coverage yes/c cluster-require-full-coverage no' /etc/redis.conf

所有6台主机都执行

mkdir -p /apps/redis/{run,data,logs,conf}# redis 配置
cat >/apps/redis/conf/redis.conf <<EOF
bind 0.0.0.0
protected-mode no
port 6379
tcp-backlog 2048
timeout 600
tcp-keepalive 300
daemonize no
supervised no
pidfile /var/run/redis_6379.pid
loglevel notice
logfile /var/log/redis.log
databases 16
always-show-logo yes
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /data
masterauth 123456
requirepass 123456
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
cluster-enabled yes
cluster-config-file nodes-6379.conf
cluster-require-full-coverage no
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
maxclients 10000
maxmemory 1gb
maxmemory-policy allkeys-lru
EOF# 安装docker-ce
yum -y install docker-ce docker-ce-cli containerd.io
systemctl enable --now docker
systemctl status docker# 加速 docker 访问
cat > /etc/docker/daemon.json <<EOF
{"registry-mirrors":["https://0cde955d3600f3000fe5c004160e0320.mirror.swr.myhuaweicloud.com"]
}
EOF# 重启 docker
systemctl restart docker# 运行容器
# WARNING: Published ports are discarded when using host network mode
docker run \
--privileged \
-p 6379:6379 \
-p 16379:16379 \
--name redis-node \
-v /apps/redis/data:/data \
-v /apps/redis/conf/redis.conf:/etc/redis/redis.conf \
--net host \
-d redis:5.0 \
redis-server /etc/redis/redis.conf# 修改配置文件属主和属组为redis
docker exec -it redis-node /bin/bash -c "chown redis.redis /etc/redis/redis.conf"

验证当前 Redis 服务状态（随机抽取一台机器）

# 开启了 16379 的 cluster 的端口,实际的端口 = redis port + 10000
[root@redis-master1 ~]# ss -ntl
State       Recv-Q Send-Q   Local Address:Port                  Peer Address:Port
LISTEN      0      128                  *:16379                            *:*
LISTEN      0      128                  *:6379                             *:*
LISTEN      0      128                  *:22                               *:*
LISTEN      0      100          127.0.0.1:25                               *:*
LISTEN      0      128               [::]:22                            [::]:*
LISTEN      0      100              [::1]:25                            [::]:*
[root@redis-master1 ~]#[root@redis-slave1 ~]# ss -ntl
State       Recv-Q Send-Q   Local Address:Port                  Peer Address:Port
LISTEN      0      128                  *:6379                             *:*
LISTEN      0      128                  *:22                               *:*
LISTEN      0      100          127.0.0.1:25                               *:*
LISTEN      0      128                  *:16379                            *:*
LISTEN      0      128               [::]:22                            [::]:*
LISTEN      0      100              [::1]:25                            [::]:*
[root@redis-slave1 ~]## 注意进程有 [cluster] 状态
[root@redis-master1 ~]# ps -ef | grep redis | grep -v grep
polkitd  24060 24039  0 18:18 ?        00:00:01 redis-server 0.0.0.0:6379 [cluster]
[root@redis-master1 ~]#[root@redis-slave3 ~]# ps -ef | grep redis | grep -v grep
polkitd   2296  2275  0 18:20 ?        00:00:00 redis-server 0.0.0.0:6379 [cluster]
[root@redis-slave3 ~]#

4.5 创建集群

# redis-cli --cluster-replicas 1 表示每个master对应一个slave节点# 在任何一台机器执行如下命令即可创建集群
redis-cli \
-a 123456 \
--cluster create \
172.18.8.17:6379 \
172.18.8.27:6379 \
172.18.8.37:6379 \
172.18.8.47:6379 \
172.18.8.57:6379 \
172.18.8.67:6379 \
--cluster-replicas 1

执行

[root@redis-master2 ~]# docker exec -it redis-node /bin/bash
root@redis-master2:/data# redis-cli \
> -a 123456 \
> --cluster create \
> 172.18.8.17:6379 \
> 172.18.8.27:6379 \
> 172.18.8.37:6379 \
> 172.18.8.47:6379 \
> 172.18.8.57:6379 \
> 172.18.8.67:6379 \
> --cluster-replicas 1
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.18.8.57:6379 to 172.18.8.17:6379
Adding replica 172.18.8.67:6379 to 172.18.8.27:6379
Adding replica 172.18.8.47:6379 to 172.18.8.37:6379
M: 07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379  # 带 M 的是 masterslots:[0-5460] (5461 slots) master                         # 当前 master 的槽位起始和结束位
M: 9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379slots:[5461-10922] (5462 slots) master
M: 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379slots:[10923-16383] (5461 slots) master
S: d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379  # 带 S 的是 slave，slave 没有槽位replicates 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
S: 61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379replicates 07813822488e7225207c935b88323b4631103d20
S: 3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379replicates 9e276df1364cd2b22987905e03cd7222670955e4
Can I set the above configuration? (type 'yes' to accept): yes       # 输入 yes 自动创建集群
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 172.18.8.17:6379)
M: 07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379slots:[0-5460] (5461 slots) master    # 已经分配的槽位1 additional replica(s)               # 分配了一个 slave
M: 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379slots:[10923-16383] (5461 slots) master1 additional replica(s)
S: 61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379slots: (0 slots) slave               # slave 没有分配槽位replicates 07813822488e7225207c935b88323b4631103d20   # 对应的 master 172.18.8.17 的 ID
S: 3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379slots: (0 slots) slavereplicates 9e276df1364cd2b22987905e03cd7222670955e4   #  对应的 master 172.18.8.27 的 ID
M: 9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379slots:[5461-10922] (5462 slots) master1 additional replica(s)
S: d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379slots: (0 slots) slavereplicates 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1   # 对应的 master 172.18.8.37 的 ID
[OK] All nodes agree about slots configuration.          # 所有节点槽位分配完成
>>> Check for open slots...                              # 检查打开的槽位
>>> Check slots coverage...                              #  检查插槽覆盖范围
[OK] All 16384 slots covered.                            #  所有槽位(16384个)分配完成
root@redis-master2:/data## 观察以上结果，可以看到3组 master / slave
172.18.8.17  172.18.8.57
172.18.8.27  172.18.8.67
172.18.8.37  172.18.8.47

4.6 查看主从状态

# Master1：172.18.8.17
[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 -c INFO replication"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Replication
role:master
connected_slaves:1
slave0:ip=172.18.8.57,port=6379,state=online,offset=6762,lag=1
master_replid:c739b886cb906f19e30de4add3e2f98d25ce368e
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:6762
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:6762
[root@redis-master1 ~]## Master2：172.18.8.27
[root@redis-master2 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 -c INFO replication"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Replication
role:master
connected_slaves:1
slave0:ip=172.18.8.67,port=6379,state=online,offset=6846,lag=1
master_replid:620317e834e3fc0200e5ce1561a028cc21666d86
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:6846
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:6846
[root@redis-master2 ~]## Master3：172.18.8.37
[root@redis-master3 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 -c INFO replication"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Replication
role:master
connected_slaves:1
slave0:ip=172.18.8.47,port=6379,state=online,offset=6860,lag=0
master_replid:8646dfa128b00251365459082d1e2c51ecd5dbbd
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:6860
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:6860
[root@redis-master3 ~]## Slave1：172.18.8.47
[root@redis-slave1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 -c INFO replication"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Replication
role:slave
master_host:172.18.8.37
master_port:6379
master_link_status:up
master_last_io_seconds_ago:8
master_sync_in_progress:0
slave_repl_offset:6902
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:8646dfa128b00251365459082d1e2c51ecd5dbbd
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:6902
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:6902
[root@redis-slave1 ~]## Slave2：172.18.8.57
[root@redis-slave2 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 -c INFO replication"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Replication
role:slave
master_host:172.18.8.17
master_port:6379
master_link_status:up
master_last_io_seconds_ago:4
master_sync_in_progress:0
slave_repl_offset:6916
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:c739b886cb906f19e30de4add3e2f98d25ce368e
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:6916
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:6916
[root@redis-slave2 ~]## Slave3：172.18.8.67
[root@redis-slave3 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 -c INFO replication"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Replication
role:slave
master_host:172.18.8.27
master_port:6379
master_link_status:up
master_last_io_seconds_ago:5
master_sync_in_progress:0
slave_repl_offset:6916
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:620317e834e3fc0200e5ce1561a028cc21666d86
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:6916
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:6916
[root@redis-slave3 ~]#

查看指定master节点的slave节点信息

# Master1：172.18.8.17
[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 cluster nodes"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379@16379 master - 0 1625746355362 3 connected 10923-16383
61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379@16379 slave 07813822488e7225207c935b88323b4631103d20 0 1625746357369 5 connected
3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379@16379 slave 9e276df1364cd2b22987905e03cd7222670955e4 0 1625746356000 6 connected
9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379@16379 master - 0 1625746356000 2 connected 5461-10922
d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379@16379 slave 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 0 1625746356366 4 connected
07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379@16379 myself,master - 0 1625746354000 1 connected 0-5460
[root@redis-master1 ~]## 以下命令查看指定 master 节点的 slave 节点信息
# 其中，0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 为 master 节点的 ID
[root@redis-master1 ~]# docker exec -it redis-node /bin/bash
root@redis-master1:/data# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> cluster slaves 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
1) "d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379@16379 slave 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 0 1625746538000 4 connected"
127.0.0.1:6379>

4.7 验证集群状态

# Master1：172.18.8.17
[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 CLUSTER INFO"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6         # 节点数
cluster_size:3                # 三个集群
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:6237
cluster_stats_messages_pong_sent:6259
cluster_stats_messages_sent:12496
cluster_stats_messages_ping_received:6254
cluster_stats_messages_pong_received:6237
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:12496
[root@redis-master1 ~]## 查看任意节点的集群状态
[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 --cluster info 172.18.8.67:6379"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.18.8.27:6379 (9e276df1...) -> 0 keys | 5462 slots | 1 slaves.
172.18.8.17:6379 (07813822...) -> 0 keys | 5461 slots | 1 slaves.
172.18.8.37:6379 (0e3572c2...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
[root@redis-master1 ~]#

4.8 查看集群 node 对应关系

[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 CLUSTER NODES"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379@16379 master - 0 1625746796000 3 connected 10923-16383
61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379@16379 slave 07813822488e7225207c935b88323b4631103d20 0 1625746797713 5 connected
3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379@16379 slave 9e276df1364cd2b22987905e03cd7222670955e4 0 1625746798000 6 connected
9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379@16379 master - 0 1625746796000 2 connected 5461-10922
d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379@16379 slave 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 0 1625746798717 4 connected
07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379@16379 myself,master - 0 1625746796000 1 connected 0-5460
[root@redis-master1 ~]#[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 --cluster check 172.18.8.67:6379"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.18.8.27:6379 (9e276df1...) -> 0 keys | 5462 slots | 1 slaves.
172.18.8.17:6379 (07813822...) -> 0 keys | 5461 slots | 1 slaves.
172.18.8.37:6379 (0e3572c2...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 172.18.8.67:6379)
S: 3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379slots: (0 slots) slavereplicates 9e276df1364cd2b22987905e03cd7222670955e4
S: 61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379slots: (0 slots) slavereplicates 07813822488e7225207c935b88323b4631103d20
M: 9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379slots:[5461-10922] (5462 slots) master1 additional replica(s)
M: 07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379slots:[0-5460] (5461 slots) master1 additional replica(s)
M: 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379slots:[10923-16383] (5461 slots) master1 additional replica(s)
S: d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379slots: (0 slots) slavereplicates 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@redis-master1 ~]#

4.9 验证集群写入key

4.9.1 redis cluster 写入key

# 经过算法计算，当前key的槽位需要写入指定的node
[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 -h 172.18.8.37 SET city1 beijing"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
(error) MOVED 3536 172.18.8.17:6379       # 槽位不在当前node所以无法写入
[root@redis-master1 ~]## 指定槽位对应 node 可写入
[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 -h 172.18.8.17 SET city1 beijing"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
OK
[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 -h 172.18.8.17 GET city1"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
"beijing"
[root@redis-master1 ~]## 172.18.8.17 对 172.18.8.57
# 对应的 slave 节点可以 KEYS *
# 但 GET city1 失败,可以到 master 上执行 GET city1
[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -h 172.18.8.57"
172.18.8.57:6379> auth 123456
OK
172.18.8.57:6379> keys *
1) "city1"
172.18.8.57:6379> get city1
(error) MOVED 3536 172.18.8.17:6379
172.18.8.57:6379>

4.9.2 redis cluster 计算 key 所属的 slot

[root@redis-master1 ~]# docker exec -it redis-node /bin/bash
root@redis-master1:/data# redis-cli -h 172.18.8.17 -a 123456 --no-auth-warning cluster nodes
0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379@16379 master - 0 1625901640822 3 connected 10923-16383
61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379@16379 slave 07813822488e7225207c935b88323b4631103d20 0 1625901642828 5 connected
d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379@16379 slave 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 0 1625901640000 4 connected
07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379@16379 myself,master - 0 1625901639000 1 connected 0-5460
3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379@16379 slave 9e276df1364cd2b22987905e03cd7222670955e4 0 1625901641824 6 connected
9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379@16379 master - 0 1625901641000 2 connected 5461-10922
root@redis-master1:/data## 计算得到 https 对应的 slot
[root@redis-master1 ~]# docker exec -it redis-node /bin/bash
root@redis-master1:/data# redis-cli -h 172.18.8.17 -a 123456 --no-auth-warning cluster keyslot https
(integer) 4368
root@redis-master1:/data# redis-cli -h 172.18.8.17 -a 123456 --no-auth-warning set https 443
OK
root@redis-master1:/data# redis-cli -h 172.18.8.17 -a 123456 --no-auth-warning cluster keyslot http
(integer) 8915
root@redis-master1:/data# redis-cli -h 172.18.8.17 -a 123456 --no-auth-warning set http 80
(error) MOVED 8915 172.18.8.27:6379
root@redis-master1:/data#[root@redis-master1 ~]# docker exec -it redis-node /bin/bash
root@redis-master1:/data# redis-cli -h 172.18.8.27 -a 123456 --no-auth-warning cluster keyslot http
(integer) 8915
root@redis-master1:/data# redis-cli -h 172.18.8.27 -a 123456 --no-auth-warning set http 80
OK
root@redis-master1:/data# redis-cli -h 172.18.8.27 -a 123456 --no-auth-warning get http
"80"
root@redis-master1:/data#

使用选项 -c 以集群模式连接

[root@redis-master1 ~]# docker exec -it redis-node /bin/bash
root@redis-master1:/data# redis-cli -c -h 172.18.8.17 -a 123456 --no-auth-warning       # 以集群模式连接
172.18.8.17:6379> cluster keyslot city2
(integer) 15795
172.18.8.17:6379> set city2 shenzhen
-> Redirected to slot [15795] located at 172.18.8.37:6379   # 转到了 Master3：172.18.8.37
OK
172.18.8.37:6379> set city3 guangzhou   # 写入到了 Master3：172.18.8.37
OK
172.18.8.37:6379> get city2
"shenzhen"
172.18.8.37:6379> get city3
"guangzhou"
172.18.8.37:6379> exit
root@redis-master1:/data# redis-cli -h 172.18.8.37 -a 123456 --no-auth-warning get city3
"guangzhou"
root@redis-master1:/data# redis-cli -h 172.18.8.37 -a 123456 --no-auth-warning get city2
"shenzhen"
root@redis-master1:/data#

4.10 python脚本实现RedisCluster集群写入

官网：https://github.com/Grokzen/redis-py-cluster

示例

[root@redis-master1 ~]# yum -y install python3
[root@redis-master1 ~]# pip3 install redis-py-cluster
[root@redis-master1 ~]# cat redis_cluster_test.py
#!/usr/bin/env python3
from rediscluster import RedisCluster
startup_nodes = [{"host":"172.18.8.17", "port":6379},{"host":"172.18.8.27", "port":6379},{"host":"172.18.8.37", "port":6379},{"host":"172.18.8.47", "port":6379},{"host":"172.18.8.57", "port":6379},{"host":"172.18.8.67", "port":6379}
]redis_conn= RedisCluster(startup_nodes=startup_nodes,password='123456',decode_responses=True)for i in range(0, 10000):redis_conn.set('key'+str(i),'value'+str(i))print('key'+str(i)+':',redis_conn.get('key'+str(i)))[root@redis-master1 ~]# chmod +x redis_cluster_test.py
[root@redis-master1 ~]# ./redis_cluster_test.py
......
key9995: value9995
key9996: value9996
key9997: value9997
key9998: value9998
key9999: value9999

验证数据

[root@redis-master1 ~]# docker exec -it redis-node /bin/bash
root@redis-master1:/data# redis-cli -a 123456 -h 172.18.8.17
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.18.8.17:6379> DBSIZE
(integer) 3333
172.18.8.17:6379> get key1
(error) MOVED 9189 172.18.8.27:6379
172.18.8.17:6379> get key2
"value2"
172.18.8.17:6379> get key3
"value3"
172.18.8.17:6379>keys *
......
3330) "key1816"
3331) "key1278"
3332) "key8322"
3333) "key5549"[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 -h 172.18.8.27 DBSIZE"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
(integer) 3341
[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 -h 172.18.8.27 GET key1"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
"value1"
[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 -h 172.18.8.37 DBSIZE"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
(integer) 3331
[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 -h 172.18.8.37 GET key45"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
"value45"
[root@redis-master1 ~]#

4.11 模拟 master 故障，对应的 slave 节点自动提升为新 master

# 模拟 Master2 节点出故障,需要相应的数秒故障转移时间
[root@redis-master2 ~]# docker exec -it redis-node /bin/bash
root@redis-master2:/data# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> shutdown
[root@redis-master2 ~]# ss -ntl
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port
LISTEN     0      128          *:111                      *:*
LISTEN     0      5      192.168.1.1:53                       *:*
LISTEN     0      128          *:22                       *:*
LISTEN     0      100    127.0.0.1:25                       *:*
LISTEN     0      128       [::]:111                   [::]:*
LISTEN     0      128       [::]:22                    [::]:*
LISTEN     0      100      [::1]:25                    [::]:*
[root@redis-master2 ~]#[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 --cluster info 172.18.8.17:6379"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Could not connect to Redis at 172.18.8.27:6379: Connection refused
172.18.8.17:6379 (07813822...) -> 3333 keys | 5461 slots | 1 slaves.
172.18.8.37:6379 (0e3572c2...) -> 3331 keys | 5461 slots | 1 slaves.
172.18.8.67:6379 (3224267b...) -> 3341 keys | 5462 slots | 0 slaves.         # 172.18.8.67 为新的 Master，没有 Slave
[OK] 10005 keys in 3 masters.
0.61 keys per slot on average.
[root@redis-master1 ~]#[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 --cluster check 172.18.8.17:6379"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Could not connect to Redis at 172.18.8.27:6379: Connection refused
172.18.8.17:6379 (07813822...) -> 3333 keys | 5461 slots | 1 slaves.
172.18.8.37:6379 (0e3572c2...) -> 3331 keys | 5461 slots | 1 slaves.
172.18.8.67:6379 (3224267b...) -> 3341 keys | 5462 slots | 0 slaves.
[OK] 10005 keys in 3 masters.
0.61 keys per slot on average.
>>> Performing Cluster Check (using node 172.18.8.17:6379)
M: 07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379slots:[0-5460] (5461 slots) master1 additional replica(s)
M: 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379slots:[10923-16383] (5461 slots) master1 additional replica(s)
S: 61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379slots: (0 slots) slavereplicates 07813822488e7225207c935b88323b4631103d20
S: d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379slots: (0 slots) slavereplicates 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
M: 3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379slots:[5461-10922] (5462 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@redis-master1 ~]#[root@redis-master1 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 -h 172.18.8.67 info replication"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Replication
role:master
connected_slaves:0
master_replid:7ed59f3aa0c933b9659b53cf0bcc5e73501eef33
master_replid2:d997daff1fc24cd84258bc156800f72f766f8c73
master_repl_offset:143598
second_repl_offset:143599
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:143598
[root@redis-master1 ~]#

恢复故障节点 Master2

[root@redis-master2 ~]# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[root@redis-master2 ~]#
[root@redis-master2 ~]# docker start  redis-node
redis-node
[root@redis-master2 ~]# docker ps
CONTAINER ID   IMAGE       COMMAND                  CREATED        STATUS         PORTS     NAMES
12e114aed3ba   redis:5.0   "docker-entrypoint.s…"   47 hours ago   Up 8 seconds             redis-node
[root@redis-master2 ~]## 查看自动生成的配置文件，可以查看 Master2 自动成为 slave 节点
[root@redis-master2 ~]# docker exec -it redis-node /bin/bash -c "cat nodes-6379.conf"
d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379@16379 slave 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 0 1625907562357 4 connected
9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379@16379 myself,slave 3224267b54957e4ca0c2ee989b2d091f4b1722a5 0 1625907562336 2 connected
3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379@16379 master - 0 1625907562357 7 connected 5461-10922
07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379@16379 master - 0 1625907562357 1 connected 0-5460
61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379@16379 slave 07813822488e7225207c935b88323b4631103d20 0 1625907562357 5 connected
0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379@16379 master - 0 1625907562357 3 connected 10923-16383
vars currentEpoch 7 lastVoteEpoch 0
[root@redis-master2 ~]#[root@redis-master2 ~]# docker exec -it redis-node /bin/bash -c "redis-cli  -a 123456 -h 172.18.8.27 info replication"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Replication
role:slave
master_host:172.18.8.67
master_port:6379
master_link_status:up
master_last_io_seconds_ago:4
master_sync_in_progress:0
slave_repl_offset:143948
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:7ed59f3aa0c933b9659b53cf0bcc5e73501eef33
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:143948
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:143599
repl_backlog_histlen:350
[root@redis-master2 ~]#

5 Redis cluster集群节点维护

redis 集群运行之后，难免由于硬件故障、网络规划、业务增长等原因对已有集群进行相应的调整，比如: 增加Redis node节点、减少节点、节点迁移、更换服务器等。增加节点和删除节点会涉及到已有的槽位重新分配及数据迁移。

5.1 集群维护之动态扩容

案例:

因公司业务发展迅猛，现有的三主三从的redis cluster架构可能无法满足现有业务的并发写入需求，因此公司紧急采购两台服务器172.18.8.77，172.18.8.87，需要将其动态添加到集群当中，但不能影响业务使用和数据丢失。

注意: 生产环境一般建议master节点为奇数个,比如:3,5,7,以防止脑裂现象

5.1.1 添加节点准备

增加Redis node节点，需要与之前的Redis node版本相同、配置一致，然后分别再启动两台Redis node，应为一主一从。

# 配置 node7 和 node8 节点
mkdir -p /apps/redis/{run,data,logs,conf}# redis 配置略，请将上面的配置文件复制即可# 安装docker-ce
yum -y install docker-ce docker-ce-cli containerd.io
systemctl enable --now docker
systemctl status docker# 加速 docker 访问
cat > /etc/docker/daemon.json <<EOF
{"registry-mirrors":["https://0cde955d3600f3000fe5c004160e0320.mirror.swr.myhuaweicloud.com"]
}
EOF# 重载配置
systemctl daemon-reload# 重启Docker
systemctl restart docker# 运行容器
# WARNING: Published ports are discarded when using host network mode
docker run \
--privileged \
-p 6379:6379 \
-p 16379:16379 \
--name redis-node \
-v /apps/redis/data:/data \
-v /apps/redis/conf/redis.conf:/etc/redis/redis.conf \
--net host \
-d redis:5.0 \
redis-server /etc/redis/redis.conf# 修改配置文件属主和属组为redis
docker exec -it redis-node /bin/bash -c "chown redis.redis /etc/redis/redis.conf"

5.1.2 添加新的master节点到集群

使用以下命令添加新节点，要添加的新redis节点IP和端口添加到的已有的集群中任意节点的IP:端口

add-node new_host:new_port existing_host:existing_port  [--slave --master-id <arg>]

说明:

header 1	header 2
new_host:new_port	为新添加的主机的IP和端口
existing_host:existing_port	为已有的集群中任意节点的IP和端口

Redis3/4 添加方式:

# 把新的 Redis 节点 172.18.8.77 添加到当前Redis集群当中。
[root@redis-node1 ~]#redis-trib.rb add-node 172.18.8.77:6379 172.17.8.87:6379
[root@redis-node1 ~]#redis-trib.rb info 172.18.8.77:6379

Redis5 添加方式

# 将一台新的主机 172.18.8.77 加入集群,以下示例中 172.18.8.27 可以是任意存在的集群节点
# redis-cli -a 123456 --cluster add-node 172.18.8.77:6379 <当前任意集群节点>:6379
[root@redis-master2 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 --cluster add-node 172.18.8.77:6379 172.18.8.27:6379"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 172.18.8.77:6379 to cluster 172.18.8.27:6379
>>> Performing Cluster Check (using node 172.18.8.27:6379)
S: 9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379slots: (0 slots) slavereplicates 3224267b54957e4ca0c2ee989b2d091f4b1722a5
S: d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379slots: (0 slots) slavereplicates 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
M: 3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379slots:[5461-10922] (5462 slots) master1 additional replica(s)
M: 07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379slots:[0-5460] (5461 slots) master1 additional replica(s)
S: 61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379slots: (0 slots) slavereplicates 07813822488e7225207c935b88323b4631103d20
M: 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379slots:[10923-16383] (5461 slots) master1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 172.18.8.77:6379 to make it join the cluster.
[OK] New node added correctly.
[root@redis-master2 ~]## 将一台新的主机 172.18.8.87 加入集群,以下示例中 172.18.8.27 可以是任意存在的集群节点
# 为新的 master 添加新的 slave 节点就必须采用下面的方法二了
[root@redis-master2 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 --cluster add-node 172.18.8.87:6379 172.18.8.27:6379"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 172.18.8.87:6379 to cluster 172.18.8.27:6379
>>> Performing Cluster Check (using node 172.18.8.27:6379)
S: 9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379slots: (0 slots) slavereplicates 3224267b54957e4ca0c2ee989b2d091f4b1722a5
S: d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379slots: (0 slots) slavereplicates 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
M: 3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379slots:[5461-10922] (5462 slots) master1 additional replica(s)
M: 07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379slots:[0-5460] (5461 slots) master1 additional replica(s)
S: 61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379slots: (0 slots) slavereplicates 07813822488e7225207c935b88323b4631103d20
M: 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379slots:[10923-16383] (5461 slots) master1 additional replica(s)
M: 9d738778c3b2b7c8b28d0911e217bc75b09d30f7 172.18.8.77:6379slots: (0 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 172.18.8.87:6379 to make it join the cluster.
[OK] New node added correctly.
[root@redis-master2 ~]#

观察到该节点已经加入成功，但此节点上没有 slot 位,也无从节点，而且新的节点是 master


[root@redis-master2 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 --cluster info 172.18.8.17:6379"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.18.8.17:6379 (07813822...) -> 3671 keys | 5471 slots | 1 slaves.
172.18.8.67:6379 (3224267b...) -> 3675 keys | 5461 slots | 1 slaves.
172.18.8.87:6379 (dde7084e...) -> 0 keys | 0 slots | 0 slaves.   # 新的节点是 master，没有从节点，也没有槽位
172.18.8.37:6379 (0e3572c2...) -> 3654 keys | 5452 slots | 1 slaves.
172.18.8.77:6379 (7e941681...) -> 0 keys | 0 slots | 0 slaves.    # 新的节点是 master，没有从节点，也没有槽位
[OK] 11000 keys in 5 masters.
0.67 keys per slot on average.
[root@redis-master2 ~]#[root@redis-master2 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 --cluster check 172.18.8.17:6379"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.18.8.17:6379 (07813822...) -> 3671 keys | 5471 slots | 1 slaves.
172.18.8.67:6379 (3224267b...) -> 3675 keys | 5461 slots | 1 slaves.
172.18.8.87:6379 (dde7084e...) -> 0 keys | 0 slots | 0 slaves.
172.18.8.37:6379 (0e3572c2...) -> 3654 keys | 5452 slots | 1 slaves.
172.18.8.77:6379 (7e941681...) -> 0 keys | 0 slots | 0 slaves.
[OK] 11000 keys in 5 masters.
0.67 keys per slot on average.
>>> Performing Cluster Check (using node 172.18.8.17:6379)
M: 07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379slots:[1365-5460],[6817-6826],[10923-12287] (5471 slots) master1 additional replica(s)
M: 3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379slots:[1356-1364],[5461-6816],[6827-10922] (5461 slots) master1 additional replica(s)
S: d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379slots: (0 slots) slavereplicates 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
M: dde7084e89d90fd368a1659012e25789176db976 172.18.8.87:6379slots: (0 slots) master
S: 9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379slots: (0 slots) slavereplicates 3224267b54957e4ca0c2ee989b2d091f4b1722a5
M: 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379slots:[0-1355],[12288-16383] (5452 slots) master1 additional replica(s)
M: 7e941681b940f68ae8969646b7e2c62f8266d308 172.18.8.77:6379slots: (0 slots) master
S: 61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379slots: (0 slots) slavereplicates 07813822488e7225207c935b88323b4631103d20
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@redis-master2 ~]#[root@redis-master2 ~]# docker exec -it redis-node /bin/bash -c "cat nodes-6379.conf"
61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379@16379 slave 07813822488e7225207c935b88323b4631103d20 0 1626342035000 12 connected
07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379@16379 master - 0 1626342040000 12 connected 1365-5460 6817-6826 10923-12287
dde7084e89d90fd368a1659012e25789176db976 172.18.8.87:6379@16379 master - 0 1626342039441 0 connected
3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379@16379 master - 0 1626342039000 11 connected 1356-1364 5461-6816 6827-10922
9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379@16379 myself,slave 3224267b54957e4ca0c2ee989b2d091f4b1722a5 0 1626342036000 2 connected
7e941681b940f68ae8969646b7e2c62f8266d308 172.18.8.77:6379@16379 master - 0 1626342038000 13 connected
0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379@16379 master - 0 1626342037433 10 connected 0-1355 12288-16383
d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379@16379 slave 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 0 1626342040443 10 connected
vars currentEpoch 13 lastVoteEpoch 0
[root@redis-master2 ~]## 和上面显示结果一样
[root@redis-master2 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 CLUSTER NODES"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379@16379 slave 07813822488e7225207c935b88323b4631103d20 0 1626342594000 12 connected
07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379@16379 master - 0 1626342597303 12 connected 1365-5460 6817-6826 10923-12287
dde7084e89d90fd368a1659012e25789176db976 172.18.8.87:6379@16379 master - 0 1626342595000 0 connected
3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379@16379 master - 0 1626342594000 11 connected 1356-1364 5461-6816 6827-10922
9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379@16379 myself,slave 3224267b54957e4ca0c2ee989b2d091f4b1722a5 0 1626342591000 2 connected
7e941681b940f68ae8969646b7e2c62f8266d308 172.18.8.77:6379@16379 master - 0 1626342595000 13 connected
0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379@16379 master - 0 1626342595297 10 connected 0-1355 12288-16383
d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379@16379 slave 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 0 1626342596300 10 connected
[root@redis-master2 ~]#

查看集群状态

[root@redis-master2 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 CLUSTER INFO"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:8
cluster_size:3
cluster_current_epoch:13
cluster_my_epoch:11
cluster_stats_messages_ping_sent:54
cluster_stats_messages_pong_sent:38
cluster_stats_messages_sent:92
cluster_stats_messages_ping_received:38
cluster_stats_messages_pong_received:36
cluster_stats_messages_received:74
[root@redis-master2 ~]#

5.1.3 在新的master上重新分配槽位

新的node节点加到集群之后,默认是master节点，但是没有slots，需要重新分配

添加主机之后需要对添加至集群种的新主机重新分片,否则其没有分片也就无法写入数据

注意: 重新分配槽位需要清空新机器的数据,所以需要先备份数据,扩展后再恢复数据（建议使用初始化之后没有使用过的新机器部署）

Redis 3/4:

[root@redis-node1 ~]# redis-trib.rb check 10.0.0.67:6379 #当前状态
[root@redis-node1 ~]# redis-trib.rb reshard <任意节点>:6379 #重新分片
[root@redis-node1 ~]# redis-trib.rb fix 10.0.0.67:6379 #如果迁移失败使用此命令修复集群

Redis 5

# redis-cli -a 123456 --cluster reshard <当前任意集群节点>:6379[root@redis-master2 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 --cluster reshard 172.18.8.17:6379"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing Cluster Check (using node 172.18.8.17:6379)
M: 07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379slots:[0-5460] (5461 slots) master1 additional replica(s)
M: 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379slots:[10923-16383] (5461 slots) master1 additional replica(s)
S: 61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379slots: (0 slots) slavereplicates 07813822488e7225207c935b88323b4631103d20
S: d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379slots: (0 slots) slavereplicates 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
M: 3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379slots:[5461-10922] (5462 slots) master1 additional replica(s)
S: 9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379slots: (0 slots) slavereplicates 3224267b54957e4ca0c2ee989b2d091f4b1722a5
M: 7e941681b940f68ae8969646b7e2c62f8266d308 172.18.8.77:6379slots: (0 slots) master
M: dde7084e89d90fd368a1659012e25789176db976 172.18.8.87:6379slots: (0 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096   # 新分配多少个槽位 = 16384 / master 个数
What is the receiving node ID? 7e941681b940f68ae8969646b7e2c62f8266d308     # 新的 master 的 ID
Please enter all the source node IDs.Type 'all' to use all the nodes as source nodes for the hash slots.Type 'done' once you entered all the source nodes IDs.
Source node #1: all   # 输入all,将哪些源主机的槽位分配给新的节点，all 是自动在所有的 redis node 选择划分，# 如果是从 redis cluster 删除某个主机可以使用此方式将指定主机上的槽位全部移动到别的 redis 主机
......
Moving slot 12285 from 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1Moving slot 12286 from 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1Moving slot 12287 from 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
Do you want to proceed with the proposed reshard plan (yes/no)? yes     # 确认分配
......
Moving slot 12285 from 172.18.8.37:6379 to 172.18.8.77:6379: .
Moving slot 12286 from 172.18.8.37:6379 to 172.18.8.77:6379:
Moving slot 12287 from 172.18.8.37:6379 to 172.18.8.77:6379:
[root@redis-master2 ~]#

确定 slot 分配成功

[root@redis-master2 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 --cluster check 172.18.8.17:6379"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.18.8.17:6379 (07813822...) -> 2747 keys | 4103 slots | 1 slaves.
172.18.8.37:6379 (0e3572c2...) -> 2740 keys | 4089 slots | 1 slaves.
172.18.8.77:6379 (7e941681...) -> 2741 keys | 4096 slots | 0 slaves.
172.18.8.87:6379 (dde7084e...) -> 0 keys | 0 slots | 0 slaves.
172.18.8.67:6379 (3224267b...) -> 2772 keys | 4096 slots | 1 slaves.
[OK] 11000 keys in 5 masters.
0.67 keys per slot on average.
>>> Performing Cluster Check (using node 172.18.8.17:6379)
M: 07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379slots:[2733-5460],[6817-6826],[10923-12287] (4103 slots) master1 additional replica(s)
S: d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379slots: (0 slots) slavereplicates 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
M: 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379slots:[12295-16383] (4089 slots) master1 additional replica(s)
M: 7e941681b940f68ae8969646b7e2c62f8266d308 172.18.8.77:6379slots:[0-2732],[5461-6816],[12288-12294] (4096 slots) master   # 可看到4096个slots，从其他原有 Master 分配过来的
S: 61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379slots: (0 slots) slavereplicates 07813822488e7225207c935b88323b4631103d20
M: dde7084e89d90fd368a1659012e25789176db976 172.18.8.87:6379slots: (0 slots) master
S: 9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379slots: (0 slots) slavereplicates 3224267b54957e4ca0c2ee989b2d091f4b1722a5
M: 3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379slots:[6827-10922] (4096 slots) master1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@redis-master2 ~]#

5.1.4 为新的master添加新的slave节点

需要再向当前的 Redis 集群中添加一个Redis单机服务器 172.18.8.87 ，用于解决当前 172.18.8.77 单机的潜在宕机问题，即，实现响应的高可用功能

有两种方式:

方法1:在新加节点到集群时，直接将之设置为slave

Redis 3/4 添加方式:

redis-trib.rb add-node --slave --master-id 7e941681b940f68ae8969646b7e2c62f8266d308 172.17.8.87:6379 <任意集群节点>:6379

Redis 5 添加方式:

redis-cli -a 123456 \
--cluster add-node 172.18.8.87:6379 <任意集群节点>:6379 \
--cluster-slave --cluster-master-id 7e941681b940f68ae8969646b7e2c62f8266d308

方法2:先将新节点加入集群，再修改为slave

为新的master添加slave节点

Redis 3/4 版本:

redis-trib.rb add-node 172.18.8.87:6379 172.18.8.17:6379

Redis 5 版本（上面已添加过，就不需要重新执行了）

# 将一台新的主机 172.18.8.87 加入集群,以下示例中 172.18.8.27 可以是任意存在的集群节点
[root@redis-master2 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 --cluster add-node 172.18.8.87:6379 172.18.8.27:6379"

更改新节点更改状态为slave:

需要手动将其指定为某个master的slave，否则其默认角色为master

[root@redis-master2 ~]# docker exec -it redis-node redis-cli -h 172.18.8.87 -p 6379 -a 123456  # 登录到新添加节点
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.18.8.87:6379> CLUSTER NODES   # 查看当前集群节点，找到目标 master 的 ID
d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379@16379 slave 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 0 1626343858000 10 connected
0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379@16379 master - 0 1626343857000 10 connected 12295-16383
07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379@16379 master - 0 1626343856000 12 connected 2733-5460 6817-6826 10923-12287
7e941681b940f68ae8969646b7e2c62f8266d308 172.18.8.77:6379@16379 master - 0 1626343859450 13 connected 0-2732 5461-6816 12288-12294
61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379@16379 slave 07813822488e7225207c935b88323b4631103d20 0 1626343858447 12 connected
9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379@16379 slave 3224267b54957e4ca0c2ee989b2d091f4b1722a5 0 1626343857444 11 connected
dde7084e89d90fd368a1659012e25789176db976 172.18.8.87:6379@16379 myself,master - 0 1626343855000 0 connected
3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379@16379 master - 0 1626343857000 11 connected 6827-10922
172.18.8.87:6379> CLUSTER REPLICATE 7e941681b940f68ae8969646b7e2c62f8266d308   # 将其设置slave，命令格式为cluster replicate MASTERID
OK
172.18.8.87:6379> CLUSTER NODES       # 再次查看集群节点状态，验证节点是否已经更改为指定 master 的 slave
d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379@16379 slave 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 0 1626343905000 10 connected
0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379@16379 master - 0 1626343906599 10 connected 12295-16383
07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379@16379 master - 0 1626343905597 12 connected 2733-5460 6817-6826 10923-12287
7e941681b940f68ae8969646b7e2c62f8266d308 172.18.8.77:6379@16379 master - 0 1626343903591 13 connected 0-2732 5461-6816 12288-12294
61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379@16379 slave 07813822488e7225207c935b88323b4631103d20 0 1626343902000 12 connected
9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379@16379 slave 3224267b54957e4ca0c2ee989b2d091f4b1722a5 0 1626343904000 11 connected
dde7084e89d90fd368a1659012e25789176db976 172.18.8.87:6379@16379 myself,slave 7e941681b940f68ae8969646b7e2c62f8266d308 0 1626343906000 0 connected
3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379@16379 master - 0 1626343905000 11 connected 6827-10922
172.18.8.87:6379>

换种方式验证

[root@redis-master2 ~]# docker exec -it redis-node /bin/bash -c "redis-cli -a 123456 --cluster check 172.18.8.17:6379"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.18.8.17:6379 (07813822...) -> 2747 keys | 4103 slots | 1 slaves.
172.18.8.37:6379 (0e3572c2...) -> 2740 keys | 4089 slots | 1 slaves.
172.18.8.77:6379 (7e941681...) -> 2741 keys | 4096 slots | 1 slaves.
172.18.8.67:6379 (3224267b...) -> 2772 keys | 4096 slots | 1 slaves.
[OK] 11000 keys in 4 masters.
0.67 keys per slot on average.
>>> Performing Cluster Check (using node 172.18.8.17:6379)
M: 07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379slots:[2733-5460],[6817-6826],[10923-12287] (4103 slots) master1 additional replica(s)
S: d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379slots: (0 slots) slavereplicates 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
M: 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379slots:[12295-16383] (4089 slots) master1 additional replica(s)
M: 7e941681b940f68ae8969646b7e2c62f8266d308 172.18.8.77:6379slots:[0-2732],[5461-6816],[12288-12294] (4096 slots) master1 additional replica(s)
S: 61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379slots: (0 slots) slavereplicates 07813822488e7225207c935b88323b4631103d20
S: dde7084e89d90fd368a1659012e25789176db976 172.18.8.87:6379slots: (0 slots) slavereplicates 7e941681b940f68ae8969646b7e2c62f8266d308
S: 9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379slots: (0 slots) slavereplicates 3224267b54957e4ca0c2ee989b2d091f4b1722a5
M: 3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379slots:[6827-10922] (4096 slots) master1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@redis-master2 ~]#[root@redis-master2 ~]# docker exec -it redis-node /bin/bash -c "redis-cli  -a 123456 -h 172.18.8.17 --no-auth-warning cluster info"
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:8  # 8 个节点
cluster_size:4         # 4 组主从
cluster_current_epoch:13
cluster_my_epoch:12
cluster_stats_messages_ping_sent:5635
cluster_stats_messages_pong_sent:5698
cluster_stats_messages_update_sent:22
cluster_stats_messages_sent:11355
cluster_stats_messages_ping_received:5698
cluster_stats_messages_pong_received:5614
cluster_stats_messages_received:11312
[root@redis-master2 ~]#

5.2 集群维护之动态缩容

案例:

由于 172.18.8.77 服务器使用年限已经超过三年，已经超过厂商质保期而且硬盘出现异常报警，经运维部架构师提交方案并与开发同事开会商议，决定将现有Redis集群的8台主服务器中的master 172.18.8.77 和对应的 slave 172.18.8.87 临时下线，三台服务器的并发写入性能足够支出未来1-2年的业务需求

删除节点过程:

添加节点的时候是先添加 node 节点到集群，然后分配槽位，删除节点的操作与添加节点的操作正好相反，是先将被要删除的 Redis node 上的槽位迁移到集群中的其他 Redis node 节点上，然后再将其删除，如果一个 Redis node 节点上的槽位没有被完全迁移，删除该 node 的时候会提示有数据且无法删除。

5.2.1 迁移 master 的槽位之其他 master

注意: 被迁移的 Redis Master 源服务器必须保证没有数据，否则迁移报错并会被强制中断。

Redis 3/4 版本

[root@redis-node1 ~]# redis-trib.rb reshard 172.18.8.77:6379
[root@redis-node1 ~]# redis-trib.rb fix 172.18.8.77:6379 #如果迁移失败使用此命令修复集群

Redis 5版本

# 查看当前状态
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -a 123456 --cluster check 172.18.8.77:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.18.8.77:6379 (9d738778...) -> 2474 keys | 4096 slots | 1 slaves.
172.18.8.17:6379 (07813822...) -> 2511 keys | 4096 slots | 1 slaves.
172.18.8.67:6379 (3224267b...) -> 2515 keys | 4096 slots | 1 slaves.
172.18.8.37:6379 (0e3572c2...) -> 2500 keys | 4096 slots | 1 slaves.
[OK] 10000 keys in 4 masters.
0.61 keys per slot on average.
>>> Performing Cluster Check (using node 172.18.8.77:6379)
M: 9d738778c3b2b7c8b28d0911e217bc75b09d30f7 172.18.8.77:6379slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master1 additional replica(s)
S: 61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379slots: (0 slots) slavereplicates 07813822488e7225207c935b88323b4631103d20
M: 07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379slots:[1365-5460] (4096 slots) master1 additional replica(s)
M: 3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379slots:[6827-10922] (4096 slots) master1 additional replica(s)
S: 9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379slots: (0 slots) slavereplicates 3224267b54957e4ca0c2ee989b2d091f4b1722a5
M: 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379slots:[12288-16383] (4096 slots) master1 additional replica(s)
S: d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379slots: (0 slots) slavereplicates 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
S: 6427d9b2781d8389a48b761f1dd3559db894ace1 172.18.8.87:6379slots: (0 slots) slavereplicates 9d738778c3b2b7c8b28d0911e217bc75b09d30f7
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@redis-master2 ~]## 连接到任意集群节点，最后1365个slot从172.18.8.77移动到第一个master节点172.18.8.37上 [root@redis-master2 ~]# docker exec -it redis-node redis-cli -a 123456 --cluster reshard 172.18.8.37:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing Cluster Check (using node 172.18.8.37:6379)
M: 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379slots:[12288-16383] (4096 slots) master1 additional replica(s)
M: 3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379slots:[6827-10922] (4096 slots) master1 additional replica(s)
M: 9d738778c3b2b7c8b28d0911e217bc75b09d30f7 172.18.8.77:6379slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master1 additional replica(s)
S: d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379slots: (0 slots) slavereplicates 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
S: 61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379slots: (0 slots) slavereplicates 07813822488e7225207c935b88323b4631103d20
S: 9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379slots: (0 slots) slavereplicates 3224267b54957e4ca0c2ee989b2d091f4b1722a5
S: 6427d9b2781d8389a48b761f1dd3559db894ace1 172.18.8.87:6379slots: (0 slots) slavereplicates 9d738778c3b2b7c8b28d0911e217bc75b09d30f7
M: 07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379slots:[1365-5460] (4096 slots) master1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 1356   # 共4096/3分别给其它三个 master节点
What is the receiving node ID? 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 # master 172.18.8.37
Please enter all the source node IDs.Type 'all' to use all the nodes as source nodes for the hash slots.Type 'done' once you entered all the source nodes IDs.
Source node #1: 9d738778c3b2b7c8b28d0911e217bc75b09d30f7 #输入要删除节点 172.18.8.77 的 ID
Source node #2: done
......
Moving slot 1354 from 9d738778c3b2b7c8b28d0911e217bc75b09d30f7
Moving slot 1355 from 9d738778c3b2b7c8b28d0911e217bc75b09d30f7
Do you want to proceed with the proposed reshard plan (yes/no)? yes
......# 非交互式方式
# 再将1365个slot从172.18.8.77移动到第二个master节点172.18.8.67上
docker exec -it redis-node /bin/bash
redis-cli \
-a 123456 \
--cluster reshard 172.18.8.67:6379 \
--cluster-slots 1365 \
--cluster-from 9d738778c3b2b7c8b28d0911e217bc75b09d30f7 \
--cluster-to 3224267b54957e4ca0c2ee989b2d091f4b1722a5 \
--cluster-yes# 非交互式方式
# 再将1365个slot从172.18.8.77移动到第二个master节点172.18.8.17上
docker exec -it redis-node /bin/bash
redis-cli \
-a 123456 \
--cluster reshard 172.18.8.17:6379 \
--cluster-slots 1375 \
--cluster-from 9d738778c3b2b7c8b28d0911e217bc75b09d30f7 \
--cluster-to 07813822488e7225207c935b88323b4631103d20 \
--cluster-yes# 确认 172.18.8.77 的所有 slot 都移走了，上面的 slave（172.18.8.87） 也自动删除，成为其它 master 的 slave
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -a 123456 --cluster check 172.18.8.77:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.18.8.77:6379 (9d738778...) -> 0 keys | 0 slots | 0 slaves.
172.18.8.17:6379 (07813822...) -> 3347 keys | 5471 slots | 2 slaves.
172.18.8.67:6379 (3224267b...) -> 3337 keys | 5461 slots | 1 slaves.
172.18.8.37:6379 (0e3572c2...) -> 3316 keys | 5452 slots | 1 slaves.
[OK] 10000 keys in 4 masters.
0.61 keys per slot on average.
>>> Performing Cluster Check (using node 172.18.8.77:6379)
M: 9d738778c3b2b7c8b28d0911e217bc75b09d30f7 172.18.8.77:6379slots: (0 slots) master    # 172.18.8.77 已经没有槽位了
S: 61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379slots: (0 slots) slavereplicates 07813822488e7225207c935b88323b4631103d20
M: 07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379slots:[1365-5460],[6817-6826],[10923-12287] (5471 slots) master2 additional replica(s)   # Master 172.18.8.17 下面有两个 Slave
M: 3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379slots:[1356-1364],[5461-6816],[6827-10922] (5461 slots) master1 additional replica(s)
S: 9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379slots: (0 slots) slavereplicates 3224267b54957e4ca0c2ee989b2d091f4b1722a5
M: 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379slots:[0-1355],[12288-16383] (5452 slots) master1 additional replica(s)
S: d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379slots: (0 slots) slavereplicates 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
S: 6427d9b2781d8389a48b761f1dd3559db894ace1 172.18.8.87:6379slots: (0 slots) slavereplicates 07813822488e7225207c935b88323b4631103d20   #  Master 从原来的 172.18.8.77 变成了 172.18.8.17
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@redis-master2 ~]## 原来 Master-172.18.8.77 的 slave-172.18.8.87
# 自动变成了 Master-172.18.8.17 的从
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -a 123456 -h 172.18.8.17 info replication
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Replication
role:master
connected_slaves:2
slave0:ip=172.18.8.57,port=6379,state=online,offset=718902,lag=1
slave1:ip=172.18.8.87,port=6379,state=online,offset=718902,lag=1
master_replid:ede57456fa3a5c8e3446c8e8a76d77465a78fb09
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:718902
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:718902
[root@redis-master2 ~]#[root@redis-master2 ~]# docker exec -it redis-node redis-cli -a 123456 -h 172.18.8.77 --no-auth-warning cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:8  # 集群中8个节点
cluster_size:3         # 少了一个主从的slot
cluster_current_epoch:12
cluster_my_epoch:9
cluster_stats_messages_ping_sent:405633
cluster_stats_messages_pong_sent:409035
cluster_stats_messages_update_sent:12
cluster_stats_messages_sent:814680
cluster_stats_messages_ping_received:409035
cluster_stats_messages_pong_received:405629
cluster_stats_messages_update_received:1
cluster_stats_messages_received:814665
[root@redis-master2 ~]#

5.2.2 从集群删除服务器

虽然槽位已经迁移完成，但是服务器IP信息还在集群当中，因此还需要将IP信息从集群删除

注意: 删除服务器前,必须清除主机上面的槽位,否则会删除主机失败

Redis 3/4:

redis-trib.rb del-node 172.18.8.77:6379 9d738778c3b2b7c8b28d0911e217bc75b09d30f7

Redis 5:

[root@redis-master2 ~]# docker exec -it redis-node redis-cli -a 123456 --cluster del-node 172.18.8.77:6379 9d738778c3b2b7c8b28d0911e217bc75b09d30f7
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Removing node 9d738778c3b2b7c8b28d0911e217bc75b09d30f7 from cluster 172.18.8.77:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@redis-master2 ~]## 删除节点后,redis进程自动关闭
# 删除节点信息
[root@master4 ~]# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500inet 172.18.8.77  netmask 255.255.0.0  broadcast 172.18.255.255inet6 fe80::250:56ff:fea3:8fd9  prefixlen 64  scopeid 0x20<link>ether 00:50:56:a3:8f:d9  txqueuelen 1000  (Ethernet)RX packets 8097495  bytes 3059389731 (2.8 GiB)RX errors 0  dropped 1504  overruns 0  frame 0TX packets 3105024  bytes 3632242580 (3.3 GiB)TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0[root@master4 ~]# ps -ef|grep redis|grep -v grep
[root@master4 ~]# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[root@master4 ~]# rm -f /apps/redis/data/nodes-6379.conf
[root@master4 ~]#
# 验证删除成功
[root@master4 ~]# ss -ntl
State      Recv-Q Send-Q Local Address:Port                Peer Address:Port
LISTEN     0      128                *:22                             *:*
LISTEN     0      100        127.0.0.1:25                             *:*
LISTEN     0      128             [::]:22                          [::]:*
LISTEN     0      100            [::1]:25                          [::]:*
[root@master4 ~]#

5.2.3 删除多余的slave节点验证结果


[root@redis-master2 ~]# docker exec -it redis-node redis-cli -a 123456 --cluster check 172.18.8.87:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.18.8.37:6379 (0e3572c2...) -> 3316 keys | 5452 slots | 1 slaves.
172.18.8.17:6379 (07813822...) -> 3347 keys | 5471 slots | 2 slaves.
172.18.8.67:6379 (3224267b...) -> 3337 keys | 5461 slots | 1 slaves.
[OK] 10000 keys in 3 masters.
0.61 keys per slot on average.
>>> Performing Cluster Check (using node 172.18.8.87:6379)
S: 6427d9b2781d8389a48b761f1dd3559db894ace1 172.18.8.87:6379slots: (0 slots) slavereplicates 07813822488e7225207c935b88323b4631103d20
S: d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379slots: (0 slots) slavereplicates 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
M: 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379slots:[0-1355],[12288-16383] (5452 slots) master1 additional replica(s)
S: 9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379slots: (0 slots) slavereplicates 3224267b54957e4ca0c2ee989b2d091f4b1722a5
M: 07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379slots:[1365-5460],[6817-6826],[10923-12287] (5471 slots) master2 additional replica(s)
S: 61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379slots: (0 slots) slavereplicates 07813822488e7225207c935b88323b4631103d20
M: 3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379slots:[1356-1364],[5461-6816],[6827-10922] (5461 slots) master1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@redis-master2 ~]## 删除多余的slave从节点
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -a 123456 --cluster del-node 172.18.8.87:6379 6427d9b2781d8389a48b761f1dd3559db894ace1
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Removing node 6427d9b2781d8389a48b761f1dd3559db894ace1 from cluster 172.18.8.87:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@redis-master2 ~]## 删除集群文件
[root@slave4 ~]# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500inet 172.18.8.87  netmask 255.255.0.0  broadcast 172.18.255.255inet6 fe80::250:56ff:fea3:2671  prefixlen 64  scopeid 0x20<link>ether 00:50:56:a3:26:71  txqueuelen 1000  (Ethernet)RX packets 8146708  bytes 3088146283 (2.8 GiB)RX errors 0  dropped 1504  overruns 0  frame 0TX packets 3136163  bytes 3715195210 (3.4 GiB)TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0[root@slave4 ~]# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[root@slave4 ~]# ps -ef|grep redis|grep -v grep
[root@slave4 ~]# rm -f /apps/redis/data/nodes-6379.conf
[root@slave4 ~]#[root@redis-master2 ~]# docker exec -it redis-node redis-cli -a 123456 --cluster check 172.18.8.17:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.18.8.17:6379 (07813822...) -> 3347 keys | 5471 slots | 1 slaves.
172.18.8.37:6379 (0e3572c2...) -> 3316 keys | 5452 slots | 1 slaves.
172.18.8.67:6379 (3224267b...) -> 3337 keys | 5461 slots | 1 slaves.
[OK] 10000 keys in 3 masters.
0.61 keys per slot on average.
>>> Performing Cluster Check (using node 172.18.8.17:6379)
M: 07813822488e7225207c935b88323b4631103d20 172.18.8.17:6379slots:[1365-5460],[6817-6826],[10923-12287] (5471 slots) master1 additional replica(s)
S: 9e276df1364cd2b22987905e03cd7222670955e4 172.18.8.27:6379slots: (0 slots) slavereplicates 3224267b54957e4ca0c2ee989b2d091f4b1722a5
M: 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1 172.18.8.37:6379slots:[0-1355],[12288-16383] (5452 slots) master1 additional replica(s)
S: d662f3ee80047246e00612fb7ce72cbf118c06b5 172.18.8.47:6379slots: (0 slots) slavereplicates 0e3572c25ada53b6a3890cf1784ac42c0ae1c1e1
M: 3224267b54957e4ca0c2ee989b2d091f4b1722a5 172.18.8.67:6379slots:[1356-1364],[5461-6816],[6827-10922] (5461 slots) master1 additional replica(s)
S: 61c9a31264e8d1940ea03993a90192475f7fb651 172.18.8.57:6379slots: (0 slots) slavereplicates 07813822488e7225207c935b88323b4631103d20
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@redis-master2 ~]#[root@redis-master2 ~]# docker exec -it redis-node redis-cli -a 123456 --cluster info 172.18.8.17:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.18.8.17:6379 (07813822...) -> 3347 keys | 5471 slots | 1 slaves.
172.18.8.37:6379 (0e3572c2...) -> 3316 keys | 5452 slots | 1 slaves.
172.18.8.67:6379 (3224267b...) -> 3337 keys | 5461 slots | 1 slaves.
[OK] 10000 keys in 3 masters.
0.61 keys per slot on average.
[root@redis-master2 ~]#

查看集群信息

[root@redis-master2 ~]# docker exec -it redis-node redis-cli -a 123456 -h 172.18.8.17 cluster info
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6  # 只有6个节点
cluster_size:3
cluster_current_epoch:12
cluster_my_epoch:12
cluster_stats_messages_ping_sent:417564
cluster_stats_messages_pong_sent:436153
cluster_stats_messages_update_sent:17
cluster_stats_messages_sent:853734
cluster_stats_messages_ping_received:436153
cluster_stats_messages_pong_received:417542
cluster_stats_messages_received:853695
[root@redis-master2 ~]#

6 集群维护之导入现有 Redis 数据至集群

官方提供了离线迁移数据到集群的工具,有些公司开发了离线迁移工具

官方工具: redis-cli --cluster import
第三方在线迁移工具: 模拟slave 节点实现, 比如: 唯品会 redis-migrate-tool , 豌豆荚 redis-port

案例:

公司将 redis cluster 部署完成之后，需要将之前的数据导入之 Redis cluster 集群，但是由于 Redis cluster 使用的分片保存 key 的机制，因此使用传统的 AOF 文件或 RDB 快照无法满足需求，因此需要使用集群数据导入命令完成

注意: 导入数据需要 redis cluster 不能与被导入的数据有重复的 key 名称，否则导入不成功或中断。

6.1 基础环境准备

导入数据之前需要关闭各 redis 服务器的密码，包括集群中的各 node 和源 Redis server ，避免认证带来
的环境不一致从而无法导入，可以加参数 --cluster-replace 强制替换 Redis cluster 已有的 key

# 在所有节点包括master和slave节点上关闭各Redis密码认证
[root@redis-master1 ~]# docker exec -it redis-node redis-cli -h 172.18.8.17 -p 6379 -a 123456 --no-auth-warning CONFIG SET requirepass ""
OK
[root@redis-master1 ~]# docker exec -it redis-node redis-cli -h 172.18.8.27 -p 6379 -a 123456 --no-auth-warning CONFIG SET requirepass ""
OK
[root@redis-master1 ~]# docker exec -it redis-node redis-cli -h 172.18.8.37 -p 6379 -a 123456 --no-auth-warning CONFIG SET requirepass ""
OK
[root@redis-master1 ~]# docker exec -it redis-node redis-cli -h 172.18.8.47 -p 6379 -a 123456 --no-auth-warning CONFIG SET requirepass ""
OK
[root@redis-master1 ~]# docker exec -it redis-node redis-cli -h 172.18.8.57 -p 6379 -a 123456 --no-auth-warning CONFIG SET requirepass ""
OK
[root@redis-master1 ~]# docker exec -it redis-node redis-cli -h 172.18.8.67 -p 6379 -a 123456 --no-auth-warning CONFIG SET requirepass ""
OK
[root@redis-master1 ~]#

6.2 执行数据导入

将源Redis server的数据直接导入之 redis cluster,此方式慎用!

Redis 3/4

redis-trib.rb import --from <外部Redis node-IP:PORT> --replace <集 群服务器IP:PORT>

Redis 5

redis-cli --cluster import <集群服务器IP:PORT> --cluster-from <外部 Redis node-IP:PORT> --cluster-copy --cluster-replace# 只使用cluster-copy，则要导入集群中的key不能存在
# 如果集群中已有同样的key，如果需要替换，可以cluster-copy和cluster-replace联用，这样集群中的 key就会被替换为外部数据

示例：将非集群节点的数据导入 redis cluster

# 在非集群节点 172.18.8.77 生成数据
[root@master4 ~]# hostname  -I|awk '{print $1}'
172.18.8.77
[root@master4 ~]## 目前没有任何数据
[root@master4 ~]# docker exec -it redis-node redis-cli -h 127.0.0.1
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> keys *
(empty list or set)
127.0.0.1:6379>[root@master4 ~]# cat redis_test_data.sh
#!/bin/bash
#
NUM=1000
PASS=123456
for i in `seq $NUM`;dodocker exec -it redis-node redis-cli -h 127.0.0.1 -a "$PASS"  --no-auth-warning  set testkey${i} testvalue${i}echo "testkey${i} testvalue${i} write done"
done
echo "$NUM key write Redis done"
[root@master4 ~]#[root@master4 ~]# bash redis_test_data.sh
......
testkey998 testvalue998 write done
OK
testkey999 testvalue999 write done
OK
testkey1000 testvalue1000 write done
1000 key write Redis done
[root@master4 ~]## 取消需要导入的主机的密码
[root@master4 ~]# docker exec -it redis-node redis-cli -h 172.18.8.77 -p 6379 -a 123456 --no-auth-warning CONFIG SET requirepass ""
OK
[root@master4 ~]## 导入数据至集群
docker exec -it redis-node \
redis-cli \
--cluster import 172.18.8.17:6379 \
--cluster-from 172.18.8.77:6379 \
--cluster-copy \
--cluster-replace# 集群现在有 11000 行数据
[root@redis-master2 ~]# docker exec -it redis-node redis-cli --cluster info 172.18.8.17:6379
172.18.8.17:6379 (07813822...) -> 3671 keys | 5471 slots | 1 slaves.
172.18.8.37:6379 (0e3572c2...) -> 3654 keys | 5452 slots | 1 slaves.
172.18.8.67:6379 (3224267b...) -> 3675 keys | 5461 slots | 1 slaves.
[OK] 11000 keys in 3 masters.
0.67 keys per slot on average.
[root@redis-master2 ~]## 验证数据
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -h 172.18.8.17 get testkey999
"testvalue999"
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -h 172.18.8.17 get testkey666
(error) MOVED 6419 172.18.8.67:6379
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -h 172.18.8.17 get testkey888
(error) MOVED 211 172.18.8.37:6379
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -h 172.18.8.37 get testkey888
"testvalue888"
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -h 172.18.8.67 get testkey666
"testvalue666"
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -h 172.18.8.67 get testkey111
(error) MOVED 13811 172.18.8.37:6379
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -h 172.18.8.37 get testkey111
"testvalue111"
[root@redis-master2 ~]## 集群模式（-c）验证数据
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -c -h 172.18.8.67 get key123      # 原来集群的数据
"value123"
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -c -h 172.18.8.67 get testkey123  # 新导入数据
"testvalue123"
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -c -h 172.18.8.67 get testkey999  # 新导入数据
"testvalue999"
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -c -h 172.18.8.67 get testkey888  # 新导入数据
"testvalue888"
[root@redis-master2 ~]# docker exec -it redis-node redis-cli -c -h 172.18.8.67 get key888     # 原来集群的数据
"value888"
[root@redis-master2 ~]#

6.3 集群偏斜

redis cluster 多个节点运行一段时间后,可能会出现倾斜现象,某个节点数据偏多,内存消耗更大,或者接受用户请求访问更多

发生倾斜的原因可能如下:

节点和槽分配不均
不同槽对应键值数量差异较大
包含bigkey,建议少用
内存相关配置不一致
热点数据不均衡 : 一致性不高时,可以使用本缓存和MQ

获取指定槽位中对应键key值的个数

redis-cli cluster countkeysinslot {slot的值}

获取指定slot对应的key个数

[root@redis-master2 ~]# docker exec -it redis-node redis-cli
127.0.0.1:6379> cluster countkeysinslot 1
(integer) 0
127.0.0.1:6379> cluster countkeysinslot 2
(integer) 0
127.0.0.1:6379> cluster countkeysinslot 3
(integer) 0
127.0.0.1:6379> cluster countkeysinslot 4
(integer) 0
127.0.0.1:6379> cluster countkeysinslot 5
(integer) 0
127.0.0.1:6379>

执行自动的槽位重新平衡分布,但会影响客户端的访问,此方法慎用

redis-cli --cluster rebalance <集群节点IP:PORT>

示例：执行自动的槽位重新平衡分布

[root@redis-master2 ~]# docker exec -it redis-node redis-cli --cluster rebalance  172.18.8.17:6379
>>> Performing Cluster Check (using node 172.18.8.17:6379)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
*** No rebalancing needed! All nodes are within the 2.00% threshold.
[root@redis-master2 ~]#

获取bigkey ,建议在slave节点执行

redis-cli --bigkeys

示例：查找 bigkey

[root@redis-master2 ~]# docker exec -it redis-node redis-cli -c --bigkeys# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).TYPE returned an error: MOVED 8241 172.18.8.67:6379
[root@redis-master2 ~]## 提示在 172.18.8.67，所以我们在 172.18.8.67 执行
[root@redis-slave3 ~]# hostname -I|awk '{print $1}'
172.18.8.67
[root@redis-slave3 ~]# docker exec -it redis-node redis-cli --bigkeys# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).[00.00%] Biggest string found so far 'testkey644' with 12 bytes
[79.97%] Biggest string found so far 'testkey1000' with 13 bytes-------- summary -------Sampled 3675 keys in the keyspace!
Total key length in bytes is 26332 (avg len 7.17)Biggest string found 'testkey1000' has 13 bytes0 lists with 0 items (00.00% of keys, avg size 0.00)
0 hashs with 0 fields (00.00% of keys, avg size 0.00)
3675 strings with 33682 bytes (100.00% of keys, avg size 9.17)
0 streams with 0 entries (00.00% of keys, avg size 0.00)
0 sets with 0 members (00.00% of keys, avg size 0.00)
0 zsets with 0 members (00.00% of keys, avg size 0.00)
[root@redis-slave3 ~]#

6.4 redis cluster 的局限性

大多数时客户端性能会“降低"
命令无法跨节点使用：mget、keys、scan、flush、sinter等
客户端维护更复杂：SDK和应用本身消耗（例如更多的连接池）
不支持多个数据库：集群模式下只有一个db 0
复制只支持一层：不支持树形复制结构,不支持级联复制
Key事务和Lua支持有限：操作的key必须在一个节点,Lua和事务无法跨节点使用

示例：跨slot的局限性

[root@redis-master2 ~]# docker exec -it redis-node redis-cli mget key1 key2 key3
(error) CROSSSLOT Keys in request don't hash to the same slot
[root@redis-master2 ~]#

7 总结

如果新机器（172.18.8.77）有数据，会导致加入集群失败

[root@redis-master2 ~]# docker exec -it redis-node redis-cli -a 123456 --cluster add-node 172.18.8.77:6379 172.18.8.27:6379

报错如下

[ERR] Node 172.18.8.77:6379 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

此时查看nodes-6379.conf会多出一行

cat  /apps/redis/data/nodes-6379.conf
# 多出一行
9379c7e9ff6c0983700e6ba017eac944af41fb37 :0@0 master,noaddr - 1625920623698 1625920622000 8 disconnected

处理办法

清除新机器 172.18.8.77 的数据

[root@master4 ~]# docker exec -it redis-node redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> flushdb
OK
127.0.0.1:6379> keys *
(empty list or set)
127.0.0.1:6379>

删除 /apps/redis/data/nodes-6379.conf行中多出的一行数据，并重启 redis 服务（所有机器都要操作）

9379c7e9ff6c0983700e6ba017eac944af41fb37 :0@0 master,noaddr - 1625920623698 1625920622000 8 disconnected