redis集群节点宕机 fail状态 redis的投票机制实测

redis集群是有很多个redis一起工作，那么就需要这个集群不是那么容易挂掉，所以呢，理论上就应该给集群中的每个节点至少一个备用的redis服务。这个备用的redis称为从节点（slave）。
.本文测试 redis集群节点宕机 fail状态以上篇博文为基础Redis集群策略及集群实例。

在Redis集群策略及集群实例中通过单机集群创建多实例（3主3从）的方式搭建了redis集群，以此为基础做实测
首先查看一下集群状态

[root@localhost redis-4.0.10]# src/redis-cli -c -h 192.168.0.108 -p 8001
192.168.0.108:8001> cluster nodes
3e79b85af9cdc988e126046f22928945285eb161 192.168.0.108:8003@18003 master - 0 1581696889554 3 connected 10923-16383
fbfaa42f64689db65dec42f5675dd37f66251371 192.168.0.108:8006@18006 slave 1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 0 1581696889553 8 connected
8706fb3d61e7ca1f433fc66534215aea8d91061e 192.168.0.108:8004@18004 slave 3e79b85af9cdc988e126046f22928945285eb161 0 1581696889000 4 connected
1924cebf2e62605d3b28a764add6b6da7e3bcbb1 192.168.0.108:8005@18005 slave 5ffc6c0d316f3352443f1979239c852e6c8e4540 0 1581696889754 5 connected
1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 192.168.0.108:8002@18002 master - 0 1581696889000 8 connected 5461-10922
76f82463d9088f62a99482818fb8690a139a60a9 192.168.0.108:8008@18008 slave 1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 0 1581696888545 8 connected
5ffc6c0d316f3352443f1979239c852e6c8e4540 192.168.0.108:8001@18001 myself,master - 0 1581696888000 1 connected 0-5460
192.168.0.108:8001>

查看控制台我们可以看出有3个主节点，4个从节点

master8001-----------slave=8005
master8002-----------slave=8006 8008
master8003-----------slave=8004

集群是如何判断是否有某个节点挂掉（宕机）

首先要说的是，每一个节点都存有这个集群所有主节点以及从节点的信息。它们之间通过互相的ping-pong判断是否节点可以连接上。如果有一半以上的节点去ping一个节点的时候没有回应，集群就认为这个节点宕机了，然后去连接它的备用节点。
由于没有更多环境测试，本文测试手动停掉主节点作为宕机测试验证

redis的投票机制

我将其中一个主节点实例服务停掉看一下他的状态
停掉8002这个主节点，它有两个从节点

192.168.0.108:8001>
[root@localhost redis-4.0.10]# ps -ef|grep redis
root      54357      1  0 Feb14 ?        00:01:21 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8001 [cluster]
root      54362      1  0 Feb14 ?        00:01:24 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8002 [cluster]
root      54367      1  0 Feb14 ?        00:01:20 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8003 [cluster]
root      54372      1  0 Feb14 ?        00:01:22 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8004 [cluster]
root      54377      1  0 Feb14 ?        00:01:21 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8005 [cluster]
root      54382      1  0 Feb14 ?        00:01:21 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8006 [cluster]
root      54649      1  0 Feb14 ?        00:01:03 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8008 [cluster]
root      56002  55813  0 00:21 pts/7    00:00:00 grep --color=auto redis
[root@localhost redis-4.0.10]# kill -9 54362

查看一下集群状态

192.168.0.108:8001> cluster nodes
3e79b85af9cdc988e126046f22928945285eb161 192.168.0.108:8003@18003 master - 0 1581697369000 3 connected 10923-16383
fbfaa42f64689db65dec42f5675dd37f66251371 192.168.0.108:8006@18006 master - 0 1581697368570 9 connected 5461-10922
8706fb3d61e7ca1f433fc66534215aea8d91061e 192.168.0.108:8004@18004 slave 3e79b85af9cdc988e126046f22928945285eb161 0 1581697370392 4 connected
1924cebf2e62605d3b28a764add6b6da7e3bcbb1 192.168.0.108:8005@18005 slave 5ffc6c0d316f3352443f1979239c852e6c8e4540 0 1581697369000 5 connected
1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 192.168.0.108:8002@18002 master,fail - 1581697316611 1581697314597 8 disconnected
76f82463d9088f62a99482818fb8690a139a60a9 192.168.0.108:8008@18008 slave fbfaa42f64689db65dec42f5675dd37f66251371 0 1581697369379 9 connected
5ffc6c0d316f3352443f1979239c852e6c8e4540 192.168.0.108:8001@18001 myself,master - 0 1581697369000 1 connected 0-5460
192.168.0.108:8001>

我们可以看到1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 192.168.0.108:8002@18002 master,fail - 1581697316611 1581697314597 8 disconnected 8002节点实例fail，那他的从节点8006，8008呢，
接着看控制到fbfaa42f64689db65dec42f5675dd37f66251371 192.168.0.108:8006@18006 master - 0 1581697368570 9 connected 5461-10922 8006从一个从节点变为新的主节点继续工作，再看800876f82463d9088f62a99482818fb8690a139a60a9 192.168.0.108:8008@18008 slave fbfaa42f64689db65dec42f5675dd37f66251371 0 1581697369379 9 connected 8008成为新的主节点8006的从节点了，
tip：集群中的每个节点都有1个至N个复制品，其中一个为主节点，其余的为从节点，如果主节点下线了，集群就会把这个主节点的一个从节点设置为新的主节点继续工作，这样集群就不会因为一个主节点的下线而无法正常工作。
三我将8002节点服务器重新启动，来看一下会有什么反应

[root@localhost redis-4.0.10]# src/redis-server /usr/local/redis-cluster/8002/redis.conf
56009:C 15 Feb 00:37:57.883 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
56009:C 15 Feb 00:37:57.883 # Redis version=4.0.10, bits=64, commit=00000000, modified=0, pid=56009, just started
56009:C 15 Feb 00:37:57.883 # Configuration loaded
[root@localhost redis-4.0.10]#

查看集群状态

[root@localhost redis-4.0.10]# src/redis-cli -c -h 192.168.0.108 -p 8001
192.168.0.108:8001> cluster nodes
3e79b85af9cdc988e126046f22928945285eb161 192.168.0.108:8003@18003 master - 0 1581698340487 3 connected 10923-16383
fbfaa42f64689db65dec42f5675dd37f66251371 192.168.0.108:8006@18006 master - 0 1581698339000 9 connected 5461-10922
8706fb3d61e7ca1f433fc66534215aea8d91061e 192.168.0.108:8004@18004 slave 3e79b85af9cdc988e126046f22928945285eb161 0 1581698339479 4 connected
1924cebf2e62605d3b28a764add6b6da7e3bcbb1 192.168.0.108:8005@18005 slave 5ffc6c0d316f3352443f1979239c852e6c8e4540 0 1581698338471 5 connected
1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 192.168.0.108:8002@18002 slave fbfaa42f64689db65dec42f5675dd37f66251371 0 1581698339479 9 connected
76f82463d9088f62a99482818fb8690a139a60a9 192.168.0.108:8008@18008 slave fbfaa42f64689db65dec42f5675dd37f66251371 0 1581698338974 9 connected
5ffc6c0d316f3352443f1979239c852e6c8e4540 192.168.0.108:8001@18001 myself,master - 0 1581698338000 1 connected 0-5460
192.168.0.108:8001>

查看控制台1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 192.168.0.108:8002@18002 slave fbfaa42f64689db65dec42f5675dd37f66251371 0 1581698339479 9 connected 原来的8002主节点成为了fbfaa42f64689db65dec42f5675dd37f66251371 192.168.0.108:8006@18006 master - 0 1581698339000 9 connected 5461-10922 新主节点8006的从节点。
tip:旧主重新连接后将其变为新主的从服务器

总结：redis的投票机制
投票过程是集群中所有master参与,如果半数以上master节点与master节点通信超时(cluster-node-timeout),认为当前master节点挂掉。
选举的依据依次是：网络连接正常->5秒内回复过INFO命令->10*down-after-milliseconds内与主连接过的->从服务器优先级->复制偏移量->运行id较小的。选出之后通过slaveif no ont将该从服务器升为新主服务器。
通过slaveof ip port命令让其他从服务器复制该信主服务器。
最后当旧主重新连接后将其变为新主的从服务器

集群进入fail状态的必要条件

1、某个主节点和所有从节点全部挂掉，我们集群就进入faill状态。
2、如果集群超过半数以上master挂掉，无论是否有slave，集群进入fail状态.
3、如果集群任意master挂掉,且当前master没有slave.集群进入fail状态
测试，我将主节点8001和它的从节点8005手动停掉，看看什么状态

[root@localhost redis-4.0.10]# ps -ef|grep redis
root      54357      1  0 Feb14 ?        00:01:28 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8001 [cluster]
root      54367      1  0 Feb14 ?        00:01:27 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8003 [cluster]
root      54372      1  0 Feb14 ?        00:01:29 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8004 [cluster]
root      54377      1  0 Feb14 ?        00:01:28 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8005 [cluster]
root      54382      1  0 Feb14 ?        00:01:28 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8006 [cluster]
root      54649      1  0 Feb14 ?        00:01:11 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8008 [cluster]
root      56010      1  0 00:37 ?        00:00:02 src/redis-server 192.168.0.108:8002 [cluster]
root      56043  55813  0 00:55 pts/7    00:00:00 grep --color=auto redis
[root@localhost redis-4.0.10]# kill -9 54357  54377
[root@localhost redis-4.0.10]#

查看服务进程

[root@localhost redis-4.0.10]# ps -ef|grep redis
root      54367      1  0 Feb14 ?        00:01:28 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8003 [cluster]
root      54372      1  0 Feb14 ?        00:01:30 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8004 [cluster]
root      54382      1  0 Feb14 ?        00:01:29 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8006 [cluster]
root      54649      1  0 Feb14 ?        00:01:11 /usr/local/redis-4.0.10/src/redis-server 192.168.0.108:8008 [cluster]
root      56010      1  0 00:37 ?        00:00:03 src/redis-server 192.168.0.108:8002 [cluster]
root      56046  55813  0 00:57 pts/7    00:00:00 grep --color=auto redis
[root@localhost redis-4.0.10]#

目前8001，8005 已关闭，查看集群状态

[root@localhost redis-4.0.10]# src/redis-cli -c -h 192.168.0.108 -p 8003
192.168.0.108:8003> cluster nodes
3e79b85af9cdc988e126046f22928945285eb161 192.168.0.108:8003@18003 myself,master - 0 1581699578000 3 connected 10923-16383
8706fb3d61e7ca1f433fc66534215aea8d91061e 192.168.0.108:8004@18004 slave 3e79b85af9cdc988e126046f22928945285eb161 0 1581699579164 4 connected
76f82463d9088f62a99482818fb8690a139a60a9 192.168.0.108:8008@18008 slave fbfaa42f64689db65dec42f5675dd37f66251371 0 1581699579000 9 connected
fbfaa42f64689db65dec42f5675dd37f66251371 192.168.0.108:8006@18006 master - 0 1581699578051 9 connected 5461-10922
5ffc6c0d316f3352443f1979239c852e6c8e4540 192.168.0.108:8001@18001 master,fail - 1581699388186 1581699386575 1 disconnected 0-5460
1924cebf2e62605d3b28a764add6b6da7e3bcbb1 192.168.0.108:8005@18005 slave,fail 5ffc6c0d316f3352443f1979239c852e6c8e4540 1581699388186 1581699386000 5 disconnected
1cd0ce2458d65b65a5a634ec287f571dbbcbb4b7 192.168.0.108:8002@18002 slave fbfaa42f64689db65dec42f5675dd37f66251371 0 1581699578000 9 connected
192.168.0.108:8003>

查看控制台5ffc6c0d316f3352443f1979239c852e6c8e4540 192.168.0.108:8001@18001 master,fail - 1581699388186 1581699386575 1 disconnected 0-5460 1924cebf2e62605d3b28a764add6b6da7e3bcbb1 192.168.0.108:8005@18005 slave,fail 5ffc6c0d316f3352443f1979239c852e6c8e4540 1581699388186 1581699386000 5 disconnected
主节点8001和从节点8005都已,fail ，8001节点redis集群就会停止工作了。

主存复制数据问题

1、如果某一个主节点和他所有的从节点都下线的话，redis集群就会停止工作了。redis集群不保证数据的强一致性，在特定的情况下，redis集群会丢失已经被执行过的写命令。
2、使用异步复制（asynchronous replication）是redis 集群可能会丢失写命令的其中一个原因，有时候由于网络原因，如果网络断开时间太长，redis集群就会启用新的主节点，之前发给主节点的数据就会丢失。

参看文档：https://www.cnblogs.com/dadonggg/p/8628735.html