kafka增加服务器,kafka增加topic的备份数量
一、困难点
建立topic的时候,可以通过指定参数 –replication-factor 设置
二、解决办法
实际上,我们可以考虑一种 “另类” 的办法:可以利用 kafka-reassign-partitions.sh 命令对所有分区进行重新分布,在做分区重新分布的时候,通过增加每个分区的replica备份数量来达到目的。
本文将介绍如何利用 kafka-reassign-partitions.sh 命令增加topic的备份数量。
注意:以下命令使用到的topic名称、zookeeper的ip和port,需要读者替换成为实际集群的参数。
(假设kafka集群有4个broker,id分别为:1001,1002,1003,1004)
2.1、获取当前topic的所有分区分布在broker的情况
[root@tbds bin]# ./kafka-topics.sh --zookeeper 172.16.32.13:2181 --topic ranger_audits --describe
Topic:ranger_audits PartitionCount:10 ReplicationFactor:1 Configs:
Topic: ranger_audits Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001
Topic: ranger_audits Partition: 1 Leader: 1002 Replicas: 1002 Isr: 1002
Topic: ranger_audits Partition: 2 Leader: 1001 Replicas: 1001 Isr: 1001
Topic: ranger_audits Partition: 3 Leader: 1002 Replicas: 1002 Isr: 1002
Topic: ranger_audits Partition: 4 Leader: 1001 Replicas: 1001 Isr: 1001
Topic: ranger_audits Partition: 5 Leader: 1002 Replicas: 1002 Isr: 1002
Topic: ranger_audits Partition: 6 Leader: 1001 Replicas: 1001 Isr: 1001
Topic: ranger_audits Partition: 7 Leader: 1002 Replicas: 1002 Isr: 1002
Topic: ranger_audits Partition: 8 Leader: 1001 Replicas: 1001 Isr: 1001
Topic: ranger_audits Partition: 9 Leader: 1002 Replicas: 1002 Isr: 1002
可以看出,ranger_audits 这个topic有10个分区,每个分区只有一个feplica备份,分布在1001和1002两台broker上面。
下面我们需要将ranger_audits 的每个分区数据都增加到2个replica备份,且分布到4个broker上面。
2.2、创建增加replica备份数量的配置文件
(注意:尽量保持topic的原有每个分区的主备份不变化。因此,配置文件的每个分区的第一个broker保持不变。)
[root@tbds bin]# vim ../config/increase-replication-factor.json
{"version":1,
"partitions":[
{"topic":"ranger_audits","partition":0,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":6,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":9,"replicas":[1002,1004]}
]}
上面的配置文件说明,我们将topic的每个分区都增加了一个replica,且保持每个分区原有的主备份所在broker不变化,将每个分区新增的replica备份数据放到到1003和1004两个broker上面。
2.3、开始执行增加分区
[root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 --reassignment-json-file ../config/increase-replication-factor.json --execute
Current partition replica assignment
{"version":1,"partitions":[{"topic":"ranger_audits","partition":3,"replicas":[1002]},{"topic":"ranger_audits","partition":9,"replicas":[1002]},{"topic":"ranger_audits","partition":8,"replicas":[1001]},{"topic":"ranger_audits","partition":1,"replicas":[1002]},{"topic":"ranger_audits","partition":4,"replicas":[1001]},{"topic":"ranger_audits","partition":2,"replicas":[1001]},{"topic":"ranger_audits","partition":5,"replicas":[1002]},{"topic":"ranger_audits","partition":0,"replicas":[1001]},{"topic":"ranger_audits","partition":6,"replicas":[1001]},{"topic":"ranger_audits","partition":7,"replicas":[1002]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":1,"partitions":[{"topic":"ranger_audits","partition":0,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":9,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":6,"replicas":[1001,1003]}]}
2.4、查看执行进度
[root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 --reassignment-json-file ../config/increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition [ranger_audits,0] completed successfully
Reassignment of partition [ranger_audits,8] completed successfully
Reassignment of partition [ranger_audits,5] completed successfully
Reassignment of partition [ranger_audits,2] completed successfully
Reassignment of partition [ranger_audits,9] completed successfully
Reassignment of partition [ranger_audits,1] completed successfully
Reassignment of partition [ranger_audits,3] completed successfully
Reassignment of partition [ranger_audits,4] completed successfully
Reassignment of partition [ranger_audits,7] completed successfully
Reassignment of partition [ranger_audits,6] completed successfully
上面显示增加分区操作成功
2.5、再次查看topic的情况
[root@tbds bin]# ./kafka-topics.sh --zookeeper 172.16.32.13:2181 --topic ranger_audits --describe
Topic:ranger_audits PartitionCount:10 ReplicationFactor:2 Configs:
Topic: ranger_audits Partition: 0 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003
Topic: ranger_audits Partition: 1 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004
Topic: ranger_audits Partition: 2 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003
Topic: ranger_audits Partition: 3 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004
Topic: ranger_audits Partition: 4 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003
Topic: ranger_audits Partition: 5 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004
Topic: ranger_audits Partition: 6 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003
Topic: ranger_audits Partition: 7 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004
Topic: ranger_audits Partition: 8 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003
Topic: ranger_audits Partition: 9 Leader: 1002 Replicas: 1002,1004 Isr: 1002,1004
从上面可以看出,备份数量增加成功
三、进一步思考
利用上述介绍的办法,除了可以用来增加topic的备份数量之外,还能够处理以下几个场景:
1、对topic的所有分区数据进行整体迁移。怎么理解呢?假如集群有N个broker,后来新扩容M个broker。由于新扩容的broker磁盘都是空的,原有的broker磁盘占用都很满。那么我们可以利用上述方法,将存储在原有N个broker的某些topic整体搬迁到新扩容的M个broker,进而实现kafka集群的整体数据均衡。
具体使用方法就是:通过编写2.2章节的配置文件,将topic的所有分区都配置到新的M个broker上面去,再执行excute,即可完成topic的所有分区数据整体迁移到新扩容的M个broker节点。
2、broker坏掉的情况。导致某些topic的某些分区的replica数量减少,可以利用kafka-reassign-partitions.sh增加replica;
3、kafka 某些broker磁盘占用很满,某些磁盘占用又很少。可以利用kafka-reassign-partitions.sh迁移某些topic的分区数据到磁盘占用少的broker,实现数据均衡;
4、kafka集群扩容。需要把原来broker的topic数据整体迁移到新的broker,合理利用新扩容的broker,实现负载均衡。
注明:本文来自投稿,不代表服务器文档网立场,如若转载,请注明出处:https://www.fwqwd.com/12212.html
kafka增加服务器,kafka增加topic的备份数量相关推荐
- Kafka精华问答 | kafka节点之间如何备份?
戳蓝字"CSDN云计算"关注我们哦! Kafka是由Apache软件基金会开发的一个开源流处理平台,由Scala和Java编写.作为一种高吞吐量的分布式发布订阅消息系统,有着诸多特 ...
- Kafka精华问答 | kafka的使用场景是什么?
戳蓝字"CSDN云计算"关注我们哦! Kafka是由Apache软件基金会开发的一个开源流处理平台,由Scala和Java编写.作为一种高吞吐量的分布式发布订阅消息系统,有着诸多特 ...
- Kafka系列之:增加Kafka节点扩展Kafka集群
Kafka系列之:增加Kafka节点扩展Kafka集群 一.增加Kafka节点 二.分区重新分配工具三种工作模式 三.自动将数据迁移到新机器 四.自定义分区分配和迁移 五.增加复制因子 六.在数据迁移 ...
- kafka tool 查看指定group下topic的堆积数量_ELK架构下利用Kafka Group实现Logstash的高可用...
系统运维的过程中,每一个细节都值得我们关注 下图为我们的基本日志处理架构 所有日志由Rsyslog或者Filebeat收集,然后传输给Kafka,Logstash作为Consumer消费Kafka里边 ...
- kafka专题:kafka的Topic主题、Partition分区、消费组偏移量offset等基本概念详解
文章目录 1. kafka集群整体架构 2. kafka相关元素的基本概念 2.1 主题Topic和分区Partition 2.2 kafka消息存储在哪里? 2.3 分区副本 2.4 消费组和偏移量 ...
- kafka集群选择多少topic和partition最合适
1. partition越多吞吐量越大 首先我们需要明白以下事实:在kafka中,单个patition是kafka并行操作的最小单元.在producer和broker端,向每一个分区写入数据是可以完全 ...
- 【kafka】服务器上Kafka启动 Cannot allocate memory
1.概述 转载:服务器上Kafka启动报错:error='Cannot allocate memory' (errno=12) 解决问题思路:大问题拆小问题.从源头(Kafka有无启动成功)开始测试, ...
- 简易kafka消息服务器搭建
分布式消息服务器kafka ##搭建过程: 步骤: 1:下载代码,解压 2:启动zookeeper服务器:bin/zookeeper-server-start.sh config/zookeeper. ...
- 如何获得云盘服务器,云服务器如何增加云盘
比如, 云盘功能开通路径: 登录西部数码网站会员管理中心-服务器管理-管理-更多-云盘管理 点击"购买云盘",确认参数无误后"立即开通"即可.(注意:每个云盘需 ...
最新文章
- 2022-2028年中国麻纺织业投资分析及前景预测报告
- linux环境内存分配原理
- python中一些常用函数和库的介绍(getattr、id、type、sys)
- android动态创建arraylist,Android:二维ArrayList帮助
- CentOS7 编译安装LVS 互为主备 (实测 笔记 Centos 7.0 + ipvsadm 1.27 + keepalived 1.2.15 )
- 网络15软工个人作业5——软件工程总结
- 商汤研究院-SpringAutoML团队招聘啦~
- 【算法】剑指 Offer 53 - I. 在排序数组中查找数字 I
- 【Siddhi】Flink Siddhi自定义函数
- 关于本博客的feed订阅
- 为什么要用dubbo,dubbo是什么,为什么要和zk结合使用?
- 大数据中心有什么作用
- chrome浏览器版本简单介绍
- 怎么可以修改pr基本图形中的文字_视频剪辑 | pr的简单教学
- 自签名证书和私有CA签名的证书的区别
- 软件系统分析模型文档
- python ddos攻击脚本_【分享】Python简易DDos攻击器源码
- 基于TLC5615的多路可调数控直流稳压电源,51单片机,含Proteus仿真和C代码等
- 两种领导力:温柔与严厉
- java this final_JAVA中的this,final,surper的用法
热门文章
- 搜出来的文本:从MCMC到模拟退火
- 直播预告 | 东南大学周张泉:基于知识图谱的推理技术
- 飞桨第三课2020.4.2
- 5个球放入3个箱子_乌龙!3个可疑箱子出现在中国总领事馆外,警方排爆后发现是口罩……...
- 【虚拟化】docker部署Rabbitmq
- jenkins自动化打包部署,jenkins执行sh脚本不退出问题
- JavaScript——易班优课YOOC课群在线测试禁止查卷解决方案
- Windows——完全控制面板(上帝模式)
- BZOJ 2733 | 洛谷 P3224 [HNOI2012]永无乡
- Number of Components