kafka节点的服役和退役

假设集群只有hadoop102、hadoop103、hadoop104中有kafka的节点，现在需要在hadoop105中新增一个节点，并且需要将部分主题的数据迁移到新增的节点中。或者将hadoop105节点退役掉，需要将105中的主题的数据迁移到现役的broker中。

服役新节点

在hadoop105上安装kafka，或者通过rsync等命令传输过去（需要注意的是通过命令传输过去的kafka需要将data、logs等删除干净）；
启动hadoop102、hadoop103、hadoop104上的kafka集群。
修改haodoop105中kafka的broker.id为3
单独启动hadoop105中的kafka

 [wyp@hadoop105 kafka]$ bin/kafka-server-start.sh -daemon ./config/server.properties

创建一个要均衡的主题

[wyp@hadoop102 kafka]$ vim topics-to-move.json {"topics": [{"topic": "first"}],"version": 1
}

生成一个负载均衡的计划

[wyp@hadoop102 kafka]$ bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092  --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2,3" --generateCurrent partition replica assignment
{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[0,2,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[2,1,0],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[1,0,2],"log_dirs":["any","any","any"]}]}Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[2,3,0],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[3,0,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[0,1,2],"log_dirs":["any","any","any"]}]}

创建副本存储计划（所有副本存储在broker0、broker1、broker2、broker3中）。

[wyp@hadoop102 kafka]$ vim increase-replication-factor.json

将上方系统生成的Proposed partition reassignment configuration下的内容复制到 increase-replication-factor.json

{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[2,3,0],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[3,0,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[0,1,2],"log_dirs":["any","any","any"]}]}

执行副本存储计划

[wyp@hadoop102 kafka]$ bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --execute

验证副本存储计划(可以通过命令的形式查看或者去105节点上查看)

[wyp@hadoop102 kafka]$ bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --verify

执行后看到如下输出就是完成节点的数据均衡了。

Status of partition reassignment:
Reassignment of partition first-0 is complete.
Reassignment of partition first-1 is complete.
Reassignment of partition first-2 is complete.Clearing broker-level throttles on brokers 0,1,2,3
Clearing topic-level throttles on topic first

退役旧节点

创建一个要均衡的主题。

[wyp@hadoop102 kafka]$ vim topics-to-move.json {"topics": [{"topic": "first"}],"version": 1
}

创建执行计划

[wyp@hadoop102 kafka]$ bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092  --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2" --generateCurrent partition replica assignment
{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[2,0,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[0,2,3],"log_dirs":["any","any","any"]}]}Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[2,0,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[0,1,2],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[1,2,0],"log_dirs":["any","any","any"]}]}

创建副本存储计划

[wyp@hadoop102 kafka]$ vim increase-replication-factor.json
# 把上方的Proposed partition reassignment configuration复制到文件中
{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[2,0,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[0,1,2],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[1,2,0],"log_dirs":["any","any","any"]}]}

执行副本存储计划

[wyp@hadoop102 kafka]$ bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --execute

验证副本存储计划

[wyp@hadoop102 kafka]$ bin/kafka-reassign-partitions.sh --bootstrap-server hadoop102:9092 --reassignment-json-file increase-replication-factor.json --verifyStatus of partition reassignment:
Reassignment of partition first-0 is complete.
Reassignment of partition first-1 is complete.
Reassignment of partition first-2 is complete.Clearing broker-level throttles on brokers 0,1,2,3
Clearing topic-level throttles on topic first

总结：
kafka节点的服役和退役的节点操作是相同，均是将主题中的数据均匀的分配到新增或者删除节点之外的borker中。

操作步骤：
(1)创建需要均衡的主题；
(2)生成负载均衡的计划；
(3)创建并执行生成的负载均衡的计划；
(4)验证计划的执行情况。

需要注意的是，服役节点时，新增的节点的brokerId需要修改，rsync复制过去的需要将data、logs中的内容删除掉。

kafka节点的服役和退役相关推荐

kafka 集群服役新节点
前言在生产环境下,kafka集群中发现遇到了性能瓶颈,主要体现在现有的集群下,某个topic的分区存储不够用了,生产者生产消息量太大,或者消费端消费速度跟不上生产进度,导致分区数据大量堆积: 在这种 ...
Kafka精华问答 | kafka节点之间如何备份？
戳蓝字"CSDN云计算"关注我们哦! Kafka是由Apache软件基金会开发的一个开源流处理平台,由Scala和Java编写.作为一种高吞吐量的分布式发布订阅消息系统,有着诸多特 ...
Kafka节点扩容、缩容后reassign partition
文章目录简介 kafka-reassign-partitions使用简介 kafka节点扩容.缩容后,需要reassign partition replica, kafka-reassign-pa ...
Kafka系列之：增加Kafka节点扩展Kafka集群
Kafka系列之:增加Kafka节点扩展Kafka集群一.增加Kafka节点二.分区重新分配工具三种工作模式三.自动将数据迁移到新机器四.自定义分区分配和迁移五.增加复制因子六.在数据迁移 ...
Kafka节点服役和退役
一.服役新节点 1. 新节点配置基础配置 vim /etc/sysconfig/network-scripts/ifcfg-ens33 BOOTPROTO=static IPADDR=192.168 ...
DataNode新节点服役与旧节点退役
大数据相关博客的目录在阅读此篇之前,应当先阅读基于CentOS7镜像和数据挂载卷实现Docker搭建Hadoop集群服役与退役 Hadoop集群中管理员经常需要向集群中添加节点,或从集群中移除节点 ...
第五天 hadoop DataNode、节点的退役与服役
DataNode和MR运行原理一.DataNode概述 <1>配置DataNode宕机时间 <2>DataNode的多目录配置二.节点的服役和退役 <1>节点的 ...
kafka(三)：kafka broker
文章目录 1. broker总体工作流程 2. Broker重要参数 3. broker节点的服役和退役 3.1 服役新节点 3.2 退役旧节点 4. kafka副本与leader选举 4.1 副本基 ...
（四）Hadoop分布式文件系统Hdfs
1.hdfs常用操作 hdfs dfs -mkdir /user/gao/ //创建文件 hdfs dfs -ls -r /user/gao //查看(递归) hdfs dfs -lsr ...

kafka节点的服役和退役

kafka节点的服役和退役

服役新节点

退役旧节点

kafka节点的服役和退役相关推荐

最新文章

热门文章