2.5：Redis Cluster 管理维护

2.5.1：动态添加节点

增加 Redis Node，需要与之前的 Redis 版本相同、配置一致；

添加一个 Redis Node 需要启动两台 Redis，一主一从：

  • 新增 Master：Redis7，192.168.1.207；
  • 新增 Slave：Redis8，192.168.1.208；

新节点配置并启动 Redis

编辑两台 Redis 服务器的 Redis 配置文件，配置 masterauth、开启集群、指定集群配置文件：

masterauth 123456
cluster-enabled yes
cluster-config-file nodes-6379.conf

启动为单机 Redis：

systemctl start redis && systemctl enable redis

添加节点到集群（Redis-3/4）

添加新的 Master 到集群中：

[root@redis1 ~]# redis-trib add-node 192.168.1.207:6379 192.168.1.201:6379 >>> Adding node 192.168.1.207:6379 to cluster 192.168.1.201:6379
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379slots:0-5460 (5461 slots) master1 additional replica(s)
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379slots: (0 slots) slavereplicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379slots:10923-16383 (5461 slots) master1 additional replica(s)
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379slots: (0 slots) slavereplicates 469e0c16495598b5fadba28f856203e206464b99
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379slots:5461-10922 (5462 slots) master1 additional replica(s)
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379slots: (0 slots) slavereplicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.1.207:6379 to make it join the cluster.
[OK] New node added correctly.

查看 Redis7 在集群当中的 ID：

Redis7 的 ID 为：4630b3bde39432a04e94414ec6b2582d26f7fc21

[root@redis1 ~]# redis-trib check 192.168.1.201:6379
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379slots:0-5460 (5461 slots) master1 additional replica(s)
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379slots: (0 slots) slavereplicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379slots:10923-16383 (5461 slots) master1 additional replica(s)
M: 4630b3bde39432a04e94414ec6b2582d26f7fc21 192.168.1.207:6379slots: (0 slots) master0 additional replica(s)
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379slots: (0 slots) slavereplicates 469e0c16495598b5fadba28f856203e206464b99
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379slots:5461-10922 (5462 slots) master1 additional replica(s)
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379slots: (0 slots) slavereplicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

添加新的 Slave 到集群中（需要指定它的 Master）：

[root@redis1 ~]# redis-trib add-node --slave --master-id 4630b3bde39432a04e94414ec6b2582d26f7fc21 192.168.1.208:6379 192.168.1.201:6379>>> Adding node 192.168.1.208:6379 to cluster 192.168.1.201:6379
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379slots:0-5460 (5461 slots) master1 additional replica(s)
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379slots: (0 slots) slavereplicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379slots:10923-16383 (5461 slots) master1 additional replica(s)
M: 4630b3bde39432a04e94414ec6b2582d26f7fc21 192.168.1.207:6379slots: (0 slots) master0 additional replica(s)
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379slots: (0 slots) slavereplicates 469e0c16495598b5fadba28f856203e206464b99
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379slots:5461-10922 (5462 slots) master1 additional replica(s)
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379slots: (0 slots) slavereplicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.1.208:6379 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 192.168.1.207:6379.
[OK] New node added correctly.

添加节点到集群（Redis-5）

添加 Master：

[root@redis1 ~]# redis-cli -a 123456 --cluster add-node 192.168.1.207:6379 192.168.1.201:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 192.168.1.207:6379 to cluster 192.168.1.201:6379
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379slots:[0-5460] (5461 slots) master1 additional replica(s)
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379slots:[10923-16383] (5461 slots) master1 additional replica(s)
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379slots:[5461-10922] (5462 slots) master1 additional replica(s)
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379slots: (0 slots) slavereplicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379slots: (0 slots) slavereplicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379slots: (0 slots) slavereplicates df921d013a6cbf1588c1ad51809435fe39f6c25c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.1.207:6379 to make it join the cluster.
[OK] New node added correctly.

查看 Redis7 在集群当中的 ID：

Redis7 的 ID：c34cbe9cba54da740cce295e318e13a1cb276d88

[root@redis2 redis5]# redis-cli -a 123456 --cluster check 192.168.1.201:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.1.201:6379 (feb6b432...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.205:6379 (df921d01...) -> 1 keys | 5461 slots | 1 slaves.
192.168.1.203:6379 (c2b48542...) -> 0 keys | 5462 slots | 1 slaves.
192.168.1.207:6379 (c34cbe9c...) -> 0 keys | 0 slots | 0 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379slots:[0-5460] (5461 slots) master1 additional replica(s)
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379slots:[10923-16383] (5461 slots) master1 additional replica(s)
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379slots:[5461-10922] (5462 slots) master1 additional replica(s)
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379slots: (0 slots) slavereplicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379slots: (0 slots) slavereplicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379slots: (0 slots) slavereplicates df921d013a6cbf1588c1ad51809435fe39f6c25c
M: c34cbe9cba54da740cce295e318e13a1cb276d88 192.168.1.207:6379slots: (0 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

添加 Slave：

[root@redis1 ~]# redis-cli -a 123456 --cluster add-node 192.168.1.208:6379 192.168.1.201:6379  --cluster-slave  --cluster-master-id c34cbe9cba54da740cce295e318e13a1cb276d88
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 192.168.1.208:6379 to cluster 192.168.1.201:6379
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379slots:[0-5460] (5461 slots) master1 additional replica(s)
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379slots:[10923-16383] (5461 slots) master1 additional replica(s)
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379slots:[5461-10922] (5462 slots) master1 additional replica(s)
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379slots: (0 slots) slavereplicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379slots: (0 slots) slavereplicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379slots: (0 slots) slavereplicates df921d013a6cbf1588c1ad51809435fe39f6c25c
M: c34cbe9cba54da740cce295e318e13a1cb276d88 192.168.1.207:6379slots: (0 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.1.208:6379 to make it join the cluster.
Waiting for the cluster to join>>> Configure node as replica of 192.168.1.207:6379.
[OK] New node added correctly.

重新分配槽位（Redis-3/4）

添加节点之后，需要重新分配集群中的槽位，否则新节点没有槽位就无法写入数据。

指定需要移动的槽位数量为 4096 个（16384 ÷ 4）；
指定接收槽位的节点为 Redis7；
指定槽位的源节点为 all；

[root@redis1 ~]# redis-trib reshard 192.168.1.201:6379How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? 4630b3bde39432a04e94414ec6b2582d26f7fc21
Please enter all the source node IDs.Type 'all' to use all the nodes as source nodes for the hash slots.Type 'done' once you entered all the source nodes IDs.
Source node #1:all

重新分配槽位（Redis-5）

[root@redis1 ~]# redis-cli -a 123456 --cluster reshard 192.168.1.201:6379How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? c34cbe9cba54da740cce295e318e13a1cb276d88
Please enter all the source node IDs.Type 'all' to use all the nodes as source nodes for the hash slots.Type 'done' once you entered all the source nodes IDs.
Source node #1: allDo you want to proceed with the proposed reshard plan (yes/no)?yes

验证集群状态

redis-trib.rb（Redis-3/4）：

[root@redis1 ~]# redis-trib info 192.168.1.201:6379
192.168.1.201:6379 (469e0c16...) -> 0 keys | 4096 slots | 1 slaves.
192.168.1.205:6379 (1b897eda...) -> 1 keys | 4096 slots | 1 slaves.
192.168.1.207:6379 (4630b3bd...) -> 0 keys | 4096 slots | 1 slaves.
192.168.1.203:6379 (4dd0b78e...) -> 0 keys | 4096 slots | 1 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.

redis-cli（Redis-5）：

[root@redis2 ~]# redis-cli -a 123456 --cluster check 192.168.1.201:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.1.201:6379 (feb6b432...) -> 0 keys | 4096 slots | 1 slaves.
192.168.1.205:6379 (df921d01...) -> 1 keys | 4096 slots | 1 slaves.
192.168.1.203:6379 (c2b48542...) -> 0 keys | 4096 slots | 1 slaves.
192.168.1.207:6379 (c34cbe9c...) -> 0 keys | 4096 slots | 1 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379slots:[1365-5460] (4096 slots) master1 additional replica(s)
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379slots:[12288-16383] (4096 slots) master1 additional replica(s)
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379slots:[6827-10922] (4096 slots) master1 additional replica(s)
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379slots: (0 slots) slavereplicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379slots: (0 slots) slavereplicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379slots: (0 slots) slavereplicates df921d013a6cbf1588c1ad51809435fe39f6c25c
S: 52724938d32b56c974249fa33a82f2d2ee54f7bb 192.168.1.208:6379slots: (0 slots) slavereplicates c34cbe9cba54da740cce295e318e13a1cb276d88
M: c34cbe9cba54da740cce295e318e13a1cb276d88 192.168.1.207:6379slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

2.5.2：动态删除节点

添加节点的时候，是先添加节点到集群，然后分配槽位；删除节点的操作与其正好相反，是先将节点上的槽位迁移到集群中的其他节点上，然后再将其删除；

如果一个节点上的槽位没有被完全迁移，是无法将其删除的。

迁移槽位（Redis-3/4）

查看要删除的节点中有多少个槽位：

这里假设要删除 Redis3（192.168.1.203），其中的槽位数量为 4096；

[root@redis1 ~]# redis-trib info 192.168.1.201:6379
192.168.1.201:6379 (469e0c16...) -> 0 keys | 4096 slots | 1 slaves.
192.168.1.205:6379 (1b897eda...) -> 1 keys | 4096 slots | 1 slaves.
192.168.1.207:6379 (4630b3bd...) -> 0 keys | 4096 slots | 1 slaves.
192.168.1.203:6379 (4dd0b78e...) -> 0 keys | 4096 slots | 1 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.

查看接收槽位的节点的 ID：

假设要将 Redis3 的槽位分配给 Redis7，记录二者的 ID：

  • Redis3：4dd0b78edebcfe93c93518b5817c00cc21bef07e

  • Redis7：4630b3bde39432a04e94414ec6b2582d26f7fc21

实际生产环境中应该是这样的场景：
Redis3 因硬件老化等原因，需要更换服务器；
此时先将新的服务器以 Master 的形式加入集群；
然后将 Redis3 的数据导出，确保槽位中无数据；
将 Redis3 的槽位迁移到新服务器；
最后移除 Redis3，向新服务器中导入数据；

[root@redis1 ~]# redis-trib check 192.168.1.201:6379
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379slots:1365-5460 (4096 slots) master1 additional replica(s)
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379slots: (0 slots) slavereplicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
S: 86664f23678b53c6adc838ede087cdca4080a85b 192.168.1.208:6379slots: (0 slots) slavereplicates 4630b3bde39432a04e94414ec6b2582d26f7fc21
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379slots:12288-16383 (4096 slots) master1 additional replica(s)
M: 4630b3bde39432a04e94414ec6b2582d26f7fc21 192.168.1.207:6379slots:0-1364,5461-6826,10923-12287 (4096 slots) master1 additional replica(s)
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379slots: (0 slots) slavereplicates 469e0c16495598b5fadba28f856203e206464b99
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379slots:6827-10922 (4096 slots) master1 additional replica(s)
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379slots: (0 slots) slavereplicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

重新分配槽位，将 Redis3 中的槽位迁移 Redis7 上：

迁移槽位前，需要保证槽位中没有数据；

[root@redis1 ~]# redis-trib reshard 192.168.1.201:6379How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? 4630b3bde39432a04e94414ec6b2582d26f7fc21
Please enter all the source node IDs.Type 'all' to use all the nodes as source nodes for the hash slots.Type 'done' once you entered all the source nodes IDs.
Source node #1:4dd0b78edebcfe93c93518b5817c00cc21bef07e
Source node #2:done

如果迁移失败，使用此命令修复集群：

redis-trib.rb fix 192.168.7101:6379

验证槽位迁移完成：

Redis3 中已无槽位；

[root@redis1 ~]# redis-trib info 192.168.1.201:6379
192.168.1.201:6379 (469e0c16...) -> 0 keys | 4096 slots | 1 slaves.
192.168.1.205:6379 (1b897eda...) -> 0 keys | 4096 slots | 1 slaves.
192.168.1.207:6379 (4630b3bd...) -> 0 keys | 8192 slots | 2 slaves.
192.168.1.203:6379 (4dd0b78e...) -> 0 keys | 0 slots | 0 slaves.
[OK] 0 keys in 4 masters.
0.00 keys per slot on average.

迁移槽位（Redis-5）

Redis3 的 ID：c2b48542344ece56e69d8ca9404ee29a48ae7b8e

Redis7 的 ID：c34cbe9cba54da740cce295e318e13a1cb276d88

[root@redis1 ~]# redis-cli -a 123456 --cluster reshard 192.168.1.201:6379How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? c34cbe9cba54da740cce295e318e13a1cb276d88
Please enter all the source node IDs.Type 'all' to use all the nodes as source nodes for the hash slots.Type 'done' once you entered all the source nodes IDs.
Source node #1: c2b48542344ece56e69d8ca9404ee29a48ae7b8e
Source node #2: doneDo you want to proceed with the proposed reshard plan (yes/no)? yes

验证槽位迁移完成：

Redis3 中已经没有槽位，而 Redis7 的槽位数位 8192；

[root@redis2 ~]# redis-cli -a 123456 --cluster check 192.168.1.201:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.1.201:6379 (feb6b432...) -> 0 keys | 4096 slots | 1 slaves.
192.168.1.205:6379 (df921d01...) -> 1 keys | 4096 slots | 1 slaves.
192.168.1.203:6379 (c2b48542...) -> 0 keys | 0 slots | 0 slaves.
192.168.1.207:6379 (c34cbe9c...) -> 0 keys | 8192 slots | 2 slaves.
[OK] 1 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379slots:[1365-5460] (4096 slots) master1 additional replica(s)
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379slots:[12288-16383] (4096 slots) master1 additional replica(s)
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379slots: (0 slots) master
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379slots: (0 slots) slavereplicates c34cbe9cba54da740cce295e318e13a1cb276d88
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379slots: (0 slots) slavereplicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379slots: (0 slots) slavereplicates df921d013a6cbf1588c1ad51809435fe39f6c25c
S: 52724938d32b56c974249fa33a82f2d2ee54f7bb 192.168.1.208:6379slots: (0 slots) slavereplicates c34cbe9cba54da740cce295e318e13a1cb276d88
M: c34cbe9cba54da740cce295e318e13a1cb276d88 192.168.1.207:6379slots:[0-1364],[5461-12287] (8192 slots) master2 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

从集群删除节点（Redis-3/4）

删除 Master：

[root@redis1 ~]# redis-trib del-node 192.168.1.201:6379 4dd0b78edebcfe93c93518b5817c00cc21bef07e
>>> Removing node 4dd0b78edebcfe93c93518b5817c00cc21bef07e from cluster 192.168.1.201:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

Master 被删除之后，其之前的 Slave 会自动成为 Redis 集群中其他 Master 的 Slave，如果不需要，也可以一并删除。

Redis7 目前有 2 个 replicas（之前 Redis3 的 Salve（Redis6）也成为 Redis7 的 Slave：

[root@redis1 ~]# redis-trib check 192.168.1.201:6379
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379
slots:2509-5460,12288-13431 (4096 slots) master
1 additional replica(s)
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379
slots: (0 slots) slave
replicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
S: 86664f23678b53c6adc838ede087cdca4080a85b 192.168.1.208:6379
slots: (0 slots) slave
replicates 4630b3bde39432a04e94414ec6b2582d26f7fc21
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379
slots:1365-2508,13432-16383 (4096 slots) master
1 additional replica(s)
M: 4630b3bde39432a04e94414ec6b2582d26f7fc21 192.168.1.207:6379
slots:0-1364,5461-12287 (8192 slots) master
2 additional replica(s)
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379
slots: (0 slots) slave
replicates 469e0c16495598b5fadba28f856203e206464b99
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379
slots: (0 slots) slave
replicates 4630b3bde39432a04e94414ec6b2582d26f7fc21
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

删除 Slave（Redis6）：

[root@redis1 ~]# redis-trib del-node 192.168.1.201:6379 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9
>>> Removing node 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 from cluster 192.168.1.201:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

从集群删除节点（Redis-5）

删除 Master：

[root@redis2 ~]# redis-cli -a 123456 --cluster del-node 192.168.1.201:6379 c2b48542344ece56e69d8ca9404ee29a48ae7b8e
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Removing node c2b48542344ece56e69d8ca9404ee29a48ae7b8e from cluster 192.168.1.201:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

删除 Slave：

Redis3 的 Slave 为 Redis6，ID为：b11dfd69bd23ae0d987a136061d7fd15b500800a

[root@redis2 ~]# redis-cli -a 123456 --cluster del-node 192.168.1.201:6379 b11dfd69bd23ae0d987a136061d7fd15b500800a
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Removing node b11dfd69bd23ae0d987a136061d7fd15b500800a from cluster 192.168.1.201:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

验证集群状态

redis-trib.rb（Redis-3/4）：

[root@redis1 ~]# redis-trib check 192.168.1.201:6379
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379slots:2509-5460,12288-13431 (4096 slots) master1 additional replica(s)
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379slots: (0 slots) slavereplicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
S: 86664f23678b53c6adc838ede087cdca4080a85b 192.168.1.208:6379slots: (0 slots) slavereplicates 4630b3bde39432a04e94414ec6b2582d26f7fc21
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379slots:1365-2508,13432-16383 (4096 slots) master1 additional replica(s)
M: 4630b3bde39432a04e94414ec6b2582d26f7fc21 192.168.1.207:6379slots:0-1364,5461-12287 (8192 slots) master1 additional replica(s)
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379slots: (0 slots) slavereplicates 469e0c16495598b5fadba28f856203e206464b99
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

redis-cli（Redis-5）：

[root@redis2 ~]# redis-cli -a 123456 --cluster check 192.168.1.201:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.1.201:6379 (feb6b432...) -> 0 keys | 4096 slots | 1 slaves.
192.168.1.205:6379 (df921d01...) -> 1 keys | 4096 slots | 1 slaves.
192.168.1.207:6379 (c34cbe9c...) -> 0 keys | 8192 slots | 1 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379slots:[1365-5460] (4096 slots) master1 additional replica(s)
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379slots:[12288-16383] (4096 slots) master1 additional replica(s)
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379slots: (0 slots) slavereplicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379slots: (0 slots) slavereplicates df921d013a6cbf1588c1ad51809435fe39f6c25c
S: 52724938d32b56c974249fa33a82f2d2ee54f7bb 192.168.1.208:6379slots: (0 slots) slavereplicates c34cbe9cba54da740cce295e318e13a1cb276d88
M: c34cbe9cba54da740cce295e318e13a1cb276d88 192.168.1.207:6379slots:[0-1364],[5461-12287] (8192 slots) master1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

2.5.3：模拟 Master 故障

测试集群中某个 Master 故障时，其 Slave 是否会自动提升为 Master 并接管业务；

测试过程：

  1. 在集群中的一台 Master 中写入测试数据，并保证 Slave 同步数据成功；
  2. 停止这台 Master 的 Redis 服务；
  3. 验证其 Slave 是否切换为 Master；
  4. 测试 Slave 切换为 Master 之后的数据读写；
  5. 重新启动原 Master 的 Redis 服务，查看集群状态；

Master 写入数据

在其中一个 Master 中写入测试数据：

在 Redis7 中写入 foo2；

[root@redis1 ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> SET foo2 bar2
(error) MOVED 1044 192.168.1.207:6379[root@redis7 ~]# redis-cli
127.0.0.1:6379> AUTH 123456
OK
127.0.0.1:6379> SET foo2 bar2
OK

验证 Slave 同步的数据：

Redis7 的 Slave 为 Redis8；

[root@redis8 ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> KEYS foo2
1) "foo2"

停止 Master 的 Redis 服务

停止 Redis7 的 Redis：

[root@redis7 ~]# systemctl stop redis

验证 Slave 切换为 Master

查看 Slave 的日志：

# 失去和 Master 的连接：
2856:S 04 Jan 18:06:32.015 # Connection with master lost.
2856:S 04 Jan 18:06:32.015 * Caching the disconnected master state.
2856:S 04 Jan 18:06:32.297 * Connecting to MASTER 192.168.1.207:6379
2856:S 04 Jan 18:06:32.298 * MASTER <-> SLAVE sync started
2856:S 04 Jan 18:06:32.298 # Error condition on socket for SYNC: Connection refused# 标记 Master 的状态为“失败”，并在延迟时间到达时开始选举（538ms后）；
2856:S 04 Jan 18:06:47.806 * Marking node 4630b3bde39432a04e94414ec6b2582d26f7fc21 as failing (quorum reached).
2856:S 04 Jan 18:06:47.806 # Cluster state changed: fail
2856:S 04 Jan 18:06:47.823 # Start of election delayed for 538 milliseconds (rank #0, offset 308).
2856:S 04 Jan 18:06:48.438 # Starting a failover election for epoch 11.# 在此同时，仍然会不断尝试连接 Master：
2856:S 04 Jan 18:07:02.975 * Connecting to MASTER 192.168.1.207:6379
2856:S 04 Jan 18:07:02.975 * MASTER <-> SLAVE sync started
2856:S 04 Jan 18:07:02.976 # Error condition on socket for SYNC: Connection refused
……# 提示目前还无法 Failover，还在等待其它节点对 Redis7 的检测结果（认为无法检测到 Redis7 的节点数量还未达到大多数）
2856:S 04 Jan 18:07:07.885 # Currently unable to failover: Waiting for votes, but majority still not reached.# 提示将在 686ms 后执行 failover 选举：
2856:S 04 Jan 18:07:48.415 # Start of election delayed for 686 milliseconds (rank #0, offset 308).
2856:S 04 Jan 18:07:48.518 # Currently unable to failover: Waiting the delay before I can start a new failover.# Salve 提升为 Master（因为只有一个 Slave，所以直接提升为 Master）：
2856:S 04 Jan 18:07:49.132 # Starting a failover election for epoch 12.
2856:S 04 Jan 18:07:49.139 # Failover election won: I'm the new master.
2856:S 04 Jan 18:07:49.139 # configEpoch set to 12 after successful failover
2856:M 04 Jan 18:07:49.139 # Setting secondary replication ID to 26954fa49f5d2c273df73d6c9ef41237c026f534, valid up to offset: 309. New replication ID is 2a72ebfe32268
3419742a55ea11b98ab36f82987
2856:M 04 Jan 18:07:49.140 * Discarding previously cached master state.

查看 Slave 的状态，已是 master 角色：

[root@redis8 ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> INFO Replication
# Replication
role:master
connected_slaves:0
master_replid:2a72ebfe322683419742a55ea11b98ab36f82987
master_replid2:26954fa49f5d2c273df73d6c9ef41237c026f534
master_repl_offset:308
second_repl_offset:309
repl_backlog_active:1
repl_backlog_size:104857600
repl_backlog_first_byte_offset:1
repl_backlog_histlen:308

查看当前集群状态：

做实验时，发现 Master 虽然切换成功，但是集群状态时 fail 的：

127.0.0.1:6379> CLUSTER INFO
cluster_state:fail
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:12
cluster_my_epoch:12
cluster_stats_messages_ping_sent:10764
cluster_stats_messages_pong_sent:3831
cluster_stats_messages_meet_sent:7
cluster_stats_messages_auth-req_sent:10
cluster_stats_messages_update_sent:7
cluster_stats_messages_sent:14619
cluster_stats_messages_ping_received:3827
cluster_stats_messages_pong_received:3725
cluster_stats_messages_fail_received:2
cluster_stats_messages_auth-ack_received:2
cluster_stats_messages_update_received:1
cluster_stats_messages_received:7557

重启 Redis8 的 Redis 进程后，日志中才出现集群 OK 的提示：

15157:M 04 Jan 18:20:18.186 # Cluster state changed: ok

还未找到集群状态切换不成功的原因；

[root@redis8 ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:12
cluster_my_epoch:12
cluster_stats_messages_ping_sent:2002
cluster_stats_messages_pong_sent:203
cluster_stats_messages_sent:2205
cluster_stats_messages_ping_received:203
cluster_stats_messages_pong_received:183
cluster_stats_messages_received:386

查看集群中的节点情况：

Redis7 的状态标记为 fail，Redis8 已为 master；

127.0.0.1:6379> CLUSTER NODES
1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379@16379 master - 0 1609755832019 9 connected 1365-2508 13432-16383
86664f23678b53c6adc838ede087cdca4080a85b 192.168.1.208:6379@16379 myself,master - 0 1609755832000 12 connected 0-1364 5461-12287
469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379@16379 master - 0 1609755833042 8 connected 2509-5460 12288-13431
4630b3bde39432a04e94414ec6b2582d26f7fc21 192.168.1.207:6379@16379 master,fail - 1609755616135 1609755616135 10 disconnected
674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379@16379 slave 1b897eda6acfb806eb27326f151e4067f0d47f7b 0 1609755831000 9 connected
6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379@16379 slave 469e0c16495598b5fadba28f856203e206464b99 0 1609755830000 8 connected

验证数据读写

Redis8 读取之前的 foo2：

127.0.0.1:6379> GET foo2
"bar2"

Redis8 写入数据：

127.0.0.1:6379> SET foo4 bar4
OK
127.0.0.1:6379> GET foo4
"bar4"

Redis8 已经可以对外提供读写服务了；

恢复原 Master 的 Redis 服务

启动 Redis7 的 Redis 进程：

[root@redis7 ~]# systemctl start redis

查看 Redis7 的 Replication 状态：

成为 Redis8 的 Slave；

127.0.0.1:6379> INFO Replication
# Replication
role:slave
master_host:192.168.1.208
master_port:6379
master_link_status:up
master_last_io_seconds_ago:3
master_sync_in_progress:0
slave_repl_offset:182
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:06b3de5832fb3bd07fccf94b78a12414083300d0
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:182
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:104857600
repl_backlog_first_byte_offset:1
repl_backlog_histlen:182

查看集群状态：

[root@redis7 ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:12
cluster_my_epoch:12
cluster_stats_messages_ping_sent:64
cluster_stats_messages_pong_sent:67
cluster_stats_messages_sent:131
cluster_stats_messages_ping_received:67
cluster_stats_messages_pong_received:64
cluster_stats_messages_update_received:2
cluster_stats_messages_received:133

查看集群中的节点情况：

cluster_stats_messages_received:133
127.0.0.1:6379> CLUSTER NODES
6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379@16379 slave 469e0c16495598b5fadba28f856203e206464b99 0 1609756105000 8 connected
4630b3bde39432a04e94414ec6b2582d26f7fc21 192.168.1.207:6379@16379 myself,slave 86664f23678b53c6adc838ede087cdca4080a85b 0 1609756102000 10 connected
1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379@16379 master - 0 1609756106000 9 connected 1365-2508 13432-16383
469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379@16379 master - 0 1609756106517 8 connected 2509-5460 12288-13431
86664f23678b53c6adc838ede087cdca4080a85b 192.168.1.208:6379@16379 master - 0 1609756107537 12 connected 0-1364 5461-12287
674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379@16379 slave 1b897eda6acfb806eb27326f151e4067f0d47f7b 0 1609756105489 9 connected

验证 Redis7 的数据：

后来添加的 foo4 也同步了过来；

127.0.0.1:6379> KEYS foo*
1) "foo2"
2) "foo4"

说明，集群中某个 Master 不可用时，其 Slaves 会通过选举来产生新的 Master，并在集群中继续对外提供服务；
当原来的 Master 重新恢复后，会成为新 Master 的 Slave，并同步目前 Master 中的数据；

2.5.4：集群导入数据

应用场景：

Redis Cluster 部署完成之后，需要将之前的单机 Redis 数据导入到 Redis Cluster，因为 Redis Cluster 使用的是分槽位保存 key 的机制，因此无法使用传统的 AOF 或 RDB 导入，需要使用集群数据导入命令。

向集群中导入数据需要 Redis Cluster 中不能有与导入数据中相同的 key，否则导入不成功或中断（可以加参数 --cluster-replace 强制替换 Redis Cluster 中已有的 key）。

准备源 Redis 数据

192.168.1.106 安装 Redis，并启动为单机 Redis：

[root@node106 ~]# systemctl start redis

写入测试数据：

[root@node106 ~]# vim redis.sh
#!/bin/bash
NUM=`seq 1 1000`
PASS='123456'
for i in ${NUM};doredis-cli -h 127.0.0.1 -a ${PASS} set key-${i} value-${i} &>/dev/null
doneecho "数据写入完成"[root@node106 ~]# bash redis.sh
数据写入完成[root@node106 ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> GET key-111
"value-111"

导入数据前的准备

导入数据之前，需要关闭集群中各节点 Redis 服务的连接密码，以及源 Redis Server 的连接密码，避免认证带来的环境不一致而无法导入。

关闭源 Redis Server 的连接密码：

[root@node106 ~]# redis-cli -a 123456 CONFIG SET requirepass ""

关闭目标 Redis Cluster 中各节点的 Redis 连接密码：

[root@redis1 ~]# redis-cli -h 192.168.1.201 -a 123456 CONFIG SET requirepass ""
[root@redis1 ~]# redis-cli -h 192.168.1.202 -a 123456 CONFIG SET requirepass ""
[root@redis1 ~]# redis-cli -h 192.168.1.204 -a 123456 CONFIG SET requirepass ""
[root@redis1 ~]# redis-cli -h 192.168.1.205 -a 123456 CONFIG SET requirepass ""
[root@redis1 ~]# redis-cli -h 192.168.1.207 -a 123456 CONFIG SET requirepass ""
[root@redis1 ~]# redis-cli -h 192.168.1.208 -a 123456 CONFIG SET requirepass ""

取消 Ruby Redis 模块配置的 Redis 连接密码：

[root@redis1 ~]# vim /usr/local/lib/ruby/gems/2.5.0/gems/redis-4.2.5/lib/redis/client.rb
class Redisclass Client# Defaults are also used for converting string keys to symbols.DEFAULTS = {……password: nil,

执行数据导入（Redis-3/4）

[root@redis1 ~]# redis-trib import --from 192.168.1.106:6379 --replace 192.168.1.201:6379

报错：

Migrating key-350 to 192.168.1.201:6379: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)
Migrating key-906 to 192.168.1.208:6379: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)
Migrating key-503 to 192.168.1.201:6379: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)
Migrating key-597 to 192.168.1.205:6379: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)
Migrating key-934 to 192.168.1.208:6379: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)
Migrating key-617 to 192.168.1.208:6379: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)

Redis-4.0.14 组成的 Redis Cluster 数据导入测试没有成功；

执行数据导入（Redis-5）

[root@redis1 ~]# redis-cli  --cluster import 192.168.1.201:6379 --cluster-from 192.168.1.106:6379 --cluster-copy

验证数据导入（Redis-5）

各节点已有 keys 存入：

[root@redis2 ~]# redis-cli  --cluster info 192.168.1.201:6379
192.168.1.201:6379 (feb6b432...) -> 249 keys | 4096 slots | 1 slaves.
192.168.1.205:6379 (df921d01...) -> 250 keys | 4096 slots | 1 slaves.
192.168.1.207:6379 (c34cbe9c...) -> 502 keys | 8192 slots | 1 slaves.
[OK] 1001 keys in 3 masters.
0.06 keys per slot on average.

读取导入的 key：

[root@redis1 ~]# redis-cli
127.0.0.1:6379> GET key-222
(error) MOVED 317 192.168.1.207:6379[root@redis7 ~]# redis-cli
127.0.0.1:6379> GET key-222
"value-222"

Redis Cluster 集群管理维护相关推荐

  1. Redis cluster集群扩容缩容原理

    1. Redis Cluster集群扩容 1.1 扩容原理 redis cluster可以实现对节点的灵活上下线控制 3个主节点分别维护自己负责的槽和对应的数据,如果希望加入一个节点实现扩容,就需要把 ...

  2. redis cluster 集群 HA 原理和实操（史上最全、面试必备）

    文章很长,建议收藏起来慢慢读!疯狂创客圈总目录 语雀版 | 总目录 码云版| 总目录 博客园版 为您奉上珍贵的学习资源 : 免费赠送 经典图书:<Java高并发核心编程(卷1)> 面试必备 ...

  3. Redis——cluster集群原理

    摘要 在 redis3.0之前,redis使用的哨兵架构,它借助 sentinel 工具来监控 master 节点的状态:如果 master 节点异常,则会做主从切换,将一台 slave 作为 mas ...

  4. Redis Cluster 集群详解

    Redis 分布式扩展之 Redis Cluster 方案 主从切换的过程中会丢失数据,因为只有一个 master,只能单点写,没有解决水平扩容的问题.而且每个节点都保存了所有数据,一个是内存的占用率 ...

  5. Ubuntu 16.04下Redis Cluster集群搭建（官方原始方案）

    前提:先安装好Redis,参考:http://www.cnblogs.com/EasonJim/p/7599941.html 说明:Redis Cluster集群模式可以做到动态增加节点和下线节点,使 ...

  6. redis cluster 集群重新启动关闭

    找遍了redis cluster官方文档,没发现有关集群重新启动和关闭的方法.为啥会没有呢,推測redis cluster至少要三个节点才干执行,三台同一时候挂掉的可能性比較小,仅仅要不同一时候挂掉. ...

  7. redis集群扩容和缩容_深入理解Redis Cluster集群

    一.背景 前面的文章<深入理解Redis哨兵机制>一文中介绍了Redis哨兵集群的工作原理,哨兵集群虽然满足了高可用的特性,但是依然存在这样的问题:即数据只能往一个主节点上进行写入. 只能 ...

  8. Redis Cluster集群的搭建与实践[转]

    Redis Cluster集群的搭建与实践 Redis Cluster集群 一.redis-cluster设计 Redis集群搭建的方式有多种,例如使用zookeeper等,但从redis 3.0之后 ...

  9. Redis Cluster集群的搭建与实践

    Redis Cluster集群 一.redis-cluster设计 Redis集群搭建的方式有多种,例如使用zookeeper等,但从redis 3.0之后版本支持redis-cluster集群,Re ...

最新文章

  1. 小程序的事件处理参数不能取得
  2. 如何有效抵抗电脑辐射
  3. 习题6-5 使用函数验证哥德巴赫猜想 (20 分)
  4. bba70_BBA的完整形式是什么？
  5. TensorFlow笔记（11） GoolgeNet
  6. Oracle Rename
  7. TensorFlow使用--MNIST分类学习入门（感知机）
  8. 对Android蓝牙UUID的理解
  9. VM虚拟机Bridge模式VMnet0网卡无法启动问题的解决
  10. 用户界面设计有效的人机交互策略_学习笔记
  11. DeeCamp2022正式开营！李开复、张亚勤亲授大师课 | 创新事
  12. “已取消到该网页的导航” chm文件无法显示错误 解决方法
  13. 2021最全 深圳互联网公司
  14. 学习java随堂练习-20220610
  15. python代码下出现红线_python踩坑系列之导入包时下划红线及报错“No module named”问题...
  16. 做旅游的就要有驴子精神
  17. java循环购物车结算系统,购物车js代码_JS实现购物车商品列表结算功能代码
  18. 计算机编程语言排行榜—TIOBE世界编程语言排行榜（2020年5月份最新版）
  19. 用身体去感受，用心去体会
  20. abap基础表之间的联系

热门文章

  1. SharePoint 2007部署过程
  2. linux远程文件拷贝方式,linux远程拷贝文件
  3. 南京大学软件工程842参考书攻略
  4. 互联网出海现在还是风口么？
  5. 学妹问我没有实际项目经验，简历要怎么写？
  6. iOS开发基础知识--碎片37
  7. 微信h5支付“网站域名ICP备案主体与商户号主体不一致”的解决方法，H5微信支付 授权函下载
  8. ABAP inner join 性能影响
  9. 钟翔平：坚持走手机浏览器架构创新之路
  10. 分布式文件系统KFS基础知识介绍