如何使用kafka增加topic的备份数量,让业务更上一层楼
本文由云+社区发表
一、困难点
建立topic的时候,可以通过指定参数 --replication-factor 设置备份数量。但是,一旦完成建立topic,则无法通过kafka-topic.sh 或者 命令修改replica数量。
二、解决办法
实际上,我们可以考虑一种 “另类” 的办法:可以利用 kafka-reassign-partitions.sh 命令对所有分区进行重新分布,在做分区重新分布的时候,通过增加每个分区的replica备份数量来达到目的。
本文将介绍如何利用 kafka-reassign-partitions.sh 命令增加topic的备份数量。
注意:以下命令使用到的topic名称、zookeeper的ip和port,需要读者替换成为实际集群的参数。
(假设kafka集群有4个broker,id分别为:1001,1002,1003,1004)
2.1、获取当前topic的所有分区分布在broker的情况
[root@tbds bin]# ./kafka-topics.sh --zookeeper 172.16.32.13:2181 --topic ranger_audits --describe
Topic:ranger_audits PartitionCount:10 ReplicationFactor:1 Configs:Topic: ranger_audits Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001Topic: ranger_audits Partition: 1 Leader: 1002 Replicas: 1002 Isr: 1002Topic: ranger_audits Partition: 2 Leader: 1001 Replicas: 1001 Isr: 1001Topic: ranger_audits Partition: 3 Leader: 1002 Replicas: 1002 Isr: 1002Topic: ranger_audits Partition: 4 Leader: 1001 Replicas: 1001 Isr: 1001Topic: ranger_audits Partition: 5 Leader: 1002 Replicas: 1002 Isr: 1002Topic: ranger_audits Partition: 6 Leader: 1001 Replicas: 1001 Isr: 1001Topic: ranger_audits Partition: 7 Leader: 1002 Replicas: 1002 Isr: 1002Topic: ranger_audits Partition: 8 Leader: 1001 Replicas: 1001 Isr: 1001Topic: ranger_audits Partition: 9 Leader: 1002 Replicas: 1002 Isr: 1002
可以看出,ranger_audits 这个topic有10个分区,每个分区只有一个feplica备份,分布在1001和1002两台broker上面。
下面我们需要将ranger_audits 的每个分区数据都增加到2个replica备份,且分布到4个broker上面。
2.2、创建增加replica备份数量的配置文件
(注意:尽量保持topic的原有每个分区的主备份不变化。因此,配置文件的每个分区的第一个broker保持不变。)
[root@tbds bin]# vim ../config/increase-replication-factor.json
{"version":1,
"partitions":[
{"topic":"ranger_audits","partition":0,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":6,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":9,"replicas":[1002,1004]}
]}
上面的配置文件说明,我们将topic的每个分区都增加了一个replica,且保持每个分区原有的主备份所在broker不变化,将每个分区新增的replica备份数据放到到1003和1004两个broker上面。
2.3、开始执行增加分区
[root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 --reassignment-json-file ../config/increase-replication-factor.json --execute
Current partition replica assignment
{"version":1,"partitions":[{"topic":"ranger_audits","partition":3,"replicas":[1002]},{"topic":"ranger_audits","partition":9,"replicas":[1002]},{"topic":"ranger_audits","partition":8,"replicas":[1001]},{"topic":"ranger_audits","partition":1,"replicas":[1002]},{"topic":"ranger_audits","partition":4,"replicas":[1001]},{"topic":"ranger_audits","partition":2,"replicas":[1001]},{"topic":"ranger_audits","partition":5,"replicas":[1002]},{"topic":"ranger_audits","partition":0,"replicas":[1001]},{"topic":"ranger_audits","partition":6,"replicas":[1001]},{"topic":"ranger_audits","partition":7,"replicas":[1002]}]}Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":1,"partitions":[{"topic":"ranger_audits","partition":0,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":9,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":6,"replicas":[1001,1003]}]}
2.4、查看执行进度
[root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 --reassignment-json-file ../config/increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition [ranger_audits,0] completed successfully
Reassignment of partition [ranger_audits,8] completed successfully
Reassignment of partition [ranger_audits,5] completed successfully
Reassignment of partition [ranger_audits,2] completed successfully
Reassignment of partition [ranger_audits,9] completed successfully
Reassignment of partition [ranger_audits,1] completed successfully
Reassignment of partition [ranger_audits,3] completed successfully
Reassignment of partition [ranger_audits,4] completed successfully
Reassignment of partition [ranger_audits,7] completed successfully
Reassignment of partition [ranger_audits,6] completed successfully
上面显示增加分区操作成功
2.5、再次查看topic的情况
[root@tbds bin]# ./kafka-topics.sh --zookeeper 172.16.32.13:2181 --topic ranger_audits --describe
Topic:ranger_audits PartitionCount:10 ReplicationFactor:2 Configs:Topic: ranger_audits Partition: 0 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003Topic: ranger_audits Partition: 1 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004Topic: ranger_audits Partition: 2 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003Topic: ranger_audits Partition: 3 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004Topic: ranger_audits Partition: 4 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003Topic: ranger_audits Partition: 5 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004Topic: ranger_audits Partition: 6 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003Topic: ranger_audits Partition: 7 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004Topic: ranger_audits Partition: 8 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003Topic: ranger_audits Partition: 9 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004
从上面可以看出,备份数量增加成功
三、进一步思考
利用上述介绍的办法,除了可以用来增加topic的备份数量之外,还能够处理以下几个场景:
1、对topic的所有分区数据进行整体迁移。怎么理解呢?假如集群有N个broker,后来新扩容M个broker。由于新扩容的broker磁盘都是空的,原有的broker磁盘占用都很满。那么我们可以利用上述方法,将存储在原有N个broker的某些topic整体搬迁到新扩容的M个broker,进而实现kafka集群的整体数据均衡。
具体使用方法就是:通过编写2.2章节的配置文件,将topic的所有分区都配置到新的M个broker上面去,再执行excute,即可完成topic的所有分区数据整体迁移到新扩容的M个broker节点。
*2、broker坏掉的情况。*导致某些topic的某些分区的replica数量减少,可以利用kafka-reassign-partitions.sh增加replica;
*3、kafka 某些broker磁盘占用很满,某些磁盘占用又很少。*可以利用kafka-reassign-partitions.sh迁移某些topic的分区数据到磁盘占用少的broker,实现数据均衡;
*4、kafka集群扩容。*需要把原来broker的topic数据整体迁移到新的broker,合理利用新扩容的broker,实现负载均衡。
此文已由作者授权腾讯云+社区在各渠道发布
获取更多新鲜技术干货,可以关注我们腾讯云技术社区-云加社区官方号及知乎机构号
转载于:https://www.cnblogs.com/qcloud1001/p/10240689.html
如何使用kafka增加topic的备份数量,让业务更上一层楼相关推荐
- kafka增加服务器,kafka增加topic的备份数量
一.困难点 建立topic的时候,可以通过指定参数 –replication-factor 设置 二.解决办法 实际上,我们可以考虑一种 "另类" 的办法:可以利用 kafka-r ...
- kafka tool 查看指定group下topic的堆积数量_ELK架构下利用Kafka Group实现Logstash的高可用...
系统运维的过程中,每一个细节都值得我们关注 下图为我们的基本日志处理架构 所有日志由Rsyslog或者Filebeat收集,然后传输给Kafka,Logstash作为Consumer消费Kafka里边 ...
- kafka专题:kafka的Topic主题、Partition分区、消费组偏移量offset等基本概念详解
文章目录 1. kafka集群整体架构 2. kafka相关元素的基本概念 2.1 主题Topic和分区Partition 2.2 kafka消息存储在哪里? 2.3 分区副本 2.4 消费组和偏移量 ...
- 052 kafka对topic的增删改查操作
一:create 1.开始使用命令 2.创建 bin/kafka-topics.sh --create --topic beifeng --zookeeper linux-hadoop01.ibeif ...
- Kafka中topic的Partition,Kafka为什么这么快,Consumer的负载均衡及consumerGroup的概念(来自学习笔记)
1.1. Kafka中topic的Partition 在Kafka文件存储中,同一个topic下有多个不同partition,每个partition为一个目录,partiton命名规则为topic ...
- Kafka精华问答 | kafka节点之间如何备份?
戳蓝字"CSDN云计算"关注我们哦! Kafka是由Apache软件基金会开发的一个开源流处理平台,由Scala和Java编写.作为一种高吞吐量的分布式发布订阅消息系统,有着诸多特 ...
- Kafka创建Topic出现replication factor: 2 larger than available brokers: 0 减少replication factor还是出现错误的解决办法
Kafka创建Topic出现replication factor: 2 larger than available brokers: 0 减少replication factor还是出现错误的解决办法 ...
- Kafka主题(Topic)的管理
主题的管理包括创建主题. 查看主题信息.修改主题和删除主题等操作.可以通过 Kafka 提供的 kafka-topics.sh 脚本来执行这些操作,这个脚本位于 $KAFKA_HOME/bin/目录下 ...
- kafka的topic操作入门
创建topic kafka-topics.sh \ --zookeeper mypc01:2181,mypc02:2181,mypc03:2181/kafka \ --create \ --topic ...
- Java kafka监控 topic的数据量count情况,每个topic的Summed Recent Offsets(总结最近的偏移量)
Java使用kafka的API来监控kafka的某些topic的数据量增量,offset,定时查总量之后,然后计算差值,然后就可以算单位间隔的每个topic的增量,kafka监控一般都是监控的吞吐量, ...
最新文章
- 基于C#的UDP协议的同步实现
- 关于同源策略的一些想法
- docker Failed to get D-Bus connection 报错
- 2.mysql数据库如何安装_MySQL数据库如何安装
- RuiJi Scraper基础 – RuiJi表达式模型
- perl学习之:肯定匹配和否定匹配
- MVC中code first方式开发,数据库的生成与更新
- H3C 双线路 nqa 联动
- 8种编程语言毕业设计参考文献大全(java,jsp,mysql,c#,asp.net,sqlserver,andorid,php)
- 关于IDEA中使用git更新代码,会覆盖掉未提交的新代码解决问题
- 标梵互动信息解说关于CSS-in-JS: 使用及优缺点
- 【错误】无法验证是否已安装所需的Microsoft更新KB2919355
- 【一种利用插值验证的FL隐私保护框架】VFL: A Verifiable Federated Learning
- 风靡厦门的一种游戏你玩过吗?竟然发明者不是郑成功,而另有其人!
- Detours 简介与简单使用
- 在UIToolbar上创建左箭头按钮(如UINavigationBar的“后退”样式)
- 【数据分发服务DDS】软件定义汽车【一】- 概述
- js实现一个在线的Excel
- libpng 源码的使用 第四节:写 (接口)
- java控件数组_java通过复选框控件数组实现添加多个复选框控件
热门文章
- asp.net 中http协议及相关知识(零碎知识记录)
- 公共网关接口CGI(Common GatewayInterface)
- 常用http状态码说明,status
- 收集WebDriver的执行命令和参数信息
- ie版本过低提示升级ie的示例
- 开源编辑器Makedown的安装
- python源码中的学习笔记_第9章_类与对象
- 快!100小时免费GPU跑起来!低价又好用的深度学习宝藏平台!
- 基于知识引入的情感分析
- 会看源码,你将比98.3%的人更容易拿到百度、阿里、腾讯、字节跳动等大公司的高薪Offer!请看这几个公众号!...