摘要

当kafka集群扩容时，新加入的节点可以很平滑的加入到集群中。但是原来创建的topic并不会将partition数据均匀的分配到这个节点上，只有新创建的topic才会分配到新的节点上。这时就可以用kafka的kafka-reassign-partitions.sh工具来进行手动平均分配。

This command moves topic partitions between replicas.
Option                                 Description
------                                 -----------
--broker-list <brokerlist>             The list of brokers to which thepartitions need to be reassigned inthe form "0,1,2". This is requiredif --topics-to-move-json-file isused to generate reassignmentconfiguration
--disable-rack-aware                   Disable rack aware replica assignment
--execute                              Kick off the reassignment as specifiedby the --reassignment-json-fileoption.
--generate                             Generate a candidate partitionreassignment configuration. Notethat this only generates a candidateassignment, it does not execute it.
--reassignment-json-file <manual       The JSON file with the partitionassignment json file path>             reassignment configurationThe formatto use is -{"partitions":[{"topic": "foo","partition": 1,"replicas": [1,2,3] }],"version":1}
--throttle <Long: throttle>            The movement of partitions will bethrottled to this value (bytes/sec).Rerunning with this option, whilst arebalance is in progress, will alterthe throttle value. The throttlerate should be at least 1 KB/s.(default: -1)
--topics-to-move-json-file <topics to  Generate a reassignment configurationreassign json file path>               to move the partitions of thespecified topics to the list ofbrokers specified by the --broker-list option. The format to use is -{"topics":[{"topic": "foo"},{"topic": "foo1"}],"version":1}
--verify                               Verify if the reassignment completedas specified by the --reassignment-json-file option. If there is athrottle engaged for the replicasspecified, and the rebalance hascompleted, the throttle will beremoved
--zookeeper <urls>                     REQUIRED: The connection string forthe zookeeper connection in the formhost:port. Multiple URLS can begiven to allow fail-over.

1.配置文件

使用kafka-reassign-partitions.sh工具前，必须先手动写一个配置文件，告诉工具你要分配哪些topic数据。

{"topics": [{"topic": "test"}],"version": 1
}

如果要移动多个topic数据可以输入多个。

2.生成分配计划

./kafka-reassign-partitions.sh --zookeeper druid1:2181 --topics-to-move-json-file topic.json  --broker-list  "0,1,2,4"  --generate

broker-list参数为brokerid，并不是broker的服务地址，执行完毕之后会生成类似下面的数据。

[root@druid1 bin]# ./kafka-reassign-partitions.sh --zookeeper druid1:2181 --topics-to-move-json-file topic.json  --broker-list  "0,1,2,4"  --generate
Current partition replica assignment{"version":1,"partitions":[{"topic":"test","partition":11,"replicas":[0,1,2]},{"topic":"test","partition":4,"replicas":[1,0,2]},{"topic":"test","partition":13,"replicas":[2,0,1]},{"topic":"test","partition":1,"replicas":[2,0,1]},{"topic":"test","partition":8,"replicas":[1,2,4,0]},{"topic":"test","partition":10,"replicas":[4,0,1,2]},{"topic":"test","partition":5,"replicas":[2,1,4,0]},{"topic":"test","partition":2,"replicas":[4,1,2,0]},{"topic":"test","partition":15,"replicas":[0,2,4,1]},{"topic":"test","partition":9,"replicas":[2,4,0,1]},{"topic":"test","partition":14,"replicas":[4,1,2,0]},{"topic":"test","partition":7,"replicas":[0,4,1,2]},{"topic":"test","partition":12,"replicas":[1,4,0,2]},{"topic":"test","partition":6,"replicas":[4,2,0,1]},{"topic":"test","partition":0,"replicas":[1,4,0,2]},{"topic":"test","partition":3,"replicas":[0,2,4,1]}]}
Proposed partition reassignment configuration{"version":1,"partitions":[{"topic":"test","partition":11,"replicas":[2,0,1]},{"topic":"test","partition":4,"replicas":[1,2,0]},{"topic":"test","partition":13,"replicas":[1,0,2]},{"topic":"test","partition":1,"replicas":[1,0,2]},{"topic":"test","partition":8,"replicas":[2,1,0]},{"topic":"test","partition":10,"replicas":[1,2,0]},{"topic":"test","partition":5,"replicas":[2,0,1]},{"topic":"test","partition":2,"replicas":[2,1,0]},{"topic":"test","partition":15,"replicas":[0,1,2]},{"topic":"test","partition":9,"replicas":[0,1,2]},{"topic":"test","partition":14,"replicas":[2,1,0]},{"topic":"test","partition":7,"replicas":[1,0,2]},{"topic":"test","partition":12,"replicas":[0,2,1]},{"topic":"test","partition":6,"replicas":[0,2,1]},{"topic":"test","partition":0,"replicas":[0,2,1]},{"topic":"test","partition":3,"replicas":[0,1,2]}]}

上面一行是topic当前partition的分配规则，下面一行是按照上面命令执行后程序自动分配的分配计划，可以根据机器实际资源情况手动调整，将确认后的分配规则写到一个文件里面，我这里命名为move.json

3.执行分配

./kafka-reassign-partitions.sh --zookeeper druid1:2181 --reassignment-json-file move.json --throttle 31457280 --execute

–throttle 为了避免迁移过程中影响kafka正常使用，这个参数可以设置迁移的流量速度，单位是byte。

4.验证/查看进度

./kafka-reassign-partitions.sh --zookeeper druid1:2181 --reassignment-json-file move.json --verify

可以查看迁移进度

其他

kafka重新分配partition相关推荐

kafka消息存储与partition副本原理(二)
消息的存储原理: 消息的文件存储机制: 前面我们知道了一个 topic 的多个 partition 在物理磁盘上的保存路径,那么我们再来分析日志的存储方式.通过 ll /tmp/kafka-logs/ ...
Kafka分区分配策略（3）——自定义分区分配策略
接上文: 1.[Kafka分区分配策略(1)--RangeAssignor] 2.[Kafka分区分配策略(2)--RoundRobinAssignor和StickyAssignor] 欢迎支持笔者新 ...
Kafka分区分配策略（2）——RoundRobinAssignor和StickyAssignor
接上文[Kafka分区分配策略(1)--RangeAssignor] 欢迎支持笔者新作:<深入理解Kafka:核心设计与实践原理>和<RabbitMQ实战指南>,同时欢迎关注笔 ...
9.Kafka 分区分配策略（Range分配策略 RoundRobin分配策略）
前言在 Kafka 实际生产过程中,每个 topic 都会有多个 partitions. 1.多个Partitions有什么好处? ①多个 partition ,能够对 broker 上的数据 ...
Kafka分区分配策略（4）——分配的实施
接上文: 1.[Kafka分区分配策略(1)--RangeAssignor] 2.[Kafka分区分配策略(2)--RoundRobinAssignor和StickyAssignor] 3.[Kafk ...
【Kafka】kafka Java api 获取 kafka topic 或者 partition 占用的磁盘大小
1.概述 kafka Java api 获取 kafka topic 或者 partition 占用的磁盘大小 package com.dtwave.kafka.storage;import org. ...
Flink消费kafka，某partition突然从头开始消费，yarn部署，无报错，很奇怪
Flink消费kafka,某partition突然从头开始消费,yarn per job部署,ui页面无报错,检查点也没有异常,很神奇,不知道什么原因?
kafka consumer、partition、rebalance
发送消息分配partition Producer发送消息到Topic时,分配partition的算法如下: 如果指定了一个partition,那么直接使用指定的partition 如果没有指定part ...
Kafka分区分配策略(Partition Assignment Strategy)
问题用过 Kafka 的同学用过都知道,每个 Topic 一般会有很多个 partitions.为了使得我们能够及时消费消息,我们也可能会启动多个 Consumer 去消费,而每个 Consumer ...

kafka重新分配partition

摘要

1.配置文件

2.生成分配计划

3.执行分配

4.验证/查看进度

其他

kafka重新分配partition相关推荐

最新文章

热门文章