Sommaire de cet article :

kafak管理常用的工具是kafka-manager和kakfa-eagle.这两个工具在分区管理上都是只管加不管减,副本调整的话,也不支持,只能自己手工操作.本文的操作也是去年双十一的沉淀,没想到今年又用了一次,索性记录下来.

这里需要一个json文件,示例如下

bash-4.3# cat increase-replication-factor.json

{

"version":1,

"partitions":[

{

"topic":"test-topic",

"partition":0,

"replicas":[

1

]

},

{

"topic":"test-topic",

"partition":1,

"replicas":[

2

]

},

{

"topic":"test-topic",

"partition":2,

"replicas":[

3

]

}

]

}

需要增加分区,就增加下面的块

{

"topic":"test-topic",

"partition":4,

"replicas":[

3

]

}

其中partition要不重复,replicas要根据节点数循环.

需要增加副本,就修改

"replicas":[

3,2

]

两个副本就有两位,三个副本就有三位.

执行修改

会返回当前的配置,记得保存一下,万一搞错了,恢复时用

bash-4.3# kafka-reassign-partitions.sh \

--zookeeper 192.168.0.1:2181 \

--reassignment-json-file increase-test-topic.json \

--execute

Current partition replica assignment

{"version":1,"partitions":[{"topic":"test-topic","partition":0,"replicas":[3],"log_dirs":["any"]},{"topic":"test-topic","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"test-topic","partition":1,"replicas":[1],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback

验证修改

需要清楚,无论分区的增加还是副本的调整都有一个过程,会有数据的移动,再进行后面的操作前要等上一个操作完成

bash-4.3# kafka-reassign-partitions.sh \

--zookeeper 192.168.0.1:2181 \

--reassignment-json-file increase-test-topic.json \

--verify

Status of partition reassignment:

Reassignment of partition test-topic-0 completed successfully

Reassignment of partition test-topic-1 completed successfully

Reassignment of partition test-topic-2 completed successfully

Partitions reassignment failed due to Partition reassignment data file is empty

kafka.common.AdminCommandFailedException: Partition reassignment data file is empty

at kafka.admin.ReassignPartitionsCommand$.parseAndValidate(ReassignPartitionsCommand.scala:266)

at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:194)

at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:190)

at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)

at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)Partitions reassignment failed due to Partition reassignment data file is empty

kafka.common.AdminCommandFailedException: Partition reassignment data file is empty

at kafka.admin.ReassignPartitionsCommand$.parseAndValidate(ReassignPartitionsCommand.scala:266)

at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:194)

at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:190)

at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)

at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)

解决:

一般是json文件格式不对,少逗号或多逗号

{

"version":1,

"partitions":[

{"topic":"test_topic","partition":0,"replicas":[1,2,3]},

{"topic":"test_topic","partition":1,"replicas":[1,2,3]},

{"topic":"test_topic","partition":2,"replicas":[1,2,3]}

{"topic":"test_topic","partition":3,"replicas":[1,2,3]}

]

}

弄了一个微信群欢迎正常学习elk的朋友扫码交流.

相关博文