解决flume整合kafka报错Attempt to heart beat failed since member id is not valid, reset it and try to re-jo

报错情景:

由flume监听文件,分类发送到kafka的两个topic,再由两个flume(f1,f2)监听这两个topic,但是报错:

原因:

f1,f2监听kafak时,因为没有配置group.id,则f1,f2默认groupId都是"flume"
,f1,f2属于同一个消费者组(consumerGroup),同一个CG中如果订阅不同的topic,每一个consumer的上下线都会造成剩余所有的 consumer 产生 reblance 行为。

解决办法:

为两个flume配置不同的group.id即可:


a3.sources.s1.type=org.apache.flume.source.kafka.KafkaSource
a3.sources.s1.kafka.bootstrap.servers=h1:9092
a3.sources.s1.kafka.topics=topic_event
a3.sources.s1.kafka.consumer.group.id=group_event


a2.sources.s1.type=org.apache.flume.source.kafka.KafkaSource
a2.sources.s1.kafka.bootstrap.servers=h1:9092
a2.sources.s1.kafka.topics=topic_start
a2.sources.s1.kafka.consumer.group.id=group_start

解决flume整合kafka报错Attempt to heart beat failed since member id is not valid, reset it and try to re-jo相关推荐

flume 对接 kafka 报错： Error while fetching metadata with correlation id 35 {=INVALID_TOPIC_EXCEPTION}
flume 对接 kafka 报错:Error while fetching metadata with correlation id 35 : {=INVALID_TOPIC_EXCEPTION} ...
spark streaming 整合kafka 报错 KafkaConsumer is not safe for multi-threaded access
问题描述 spark streaming 使用直连方式读取kafka 数据,使用窗口时出现 java.util.ConcurrentModificationException: KafkaCons ...
【问题解决】Kafka报错 Bootstrap broker x.x.x.x:9092 (id: -1 rack: null) disconnected
问题复现近日针对某一客户需求开发了一个需要使用Kafka的功能,功能是什么暂且不论,在本地虚机的Kafka连接一切正常遂放到测试服务器上验证功能,以下是监听topic成功和警告报错: 2023-05 ...
解决idea开启tomcat报错问题 Configuration Error: deployment source ‘untitled:war exploded‘ is not valid
在idea搭建环境时遇到了这个问题报错如下: 解决方法如下:打开Project Settings->Artifacts->点+号->javaFx Application->Fr ...
kafka 报错：kafka server：Message was too large ，server rejected it to avoid allocation
kafka报错信息: kafka server:Message was too large ,server rejected it to avoid allocation kafka版本: kafk ...
命令消费kafka报错(id: -1 rack: null) disconnected
CDP 717环境使用kafka-console-consumer 命令消费kafka报错如下 23/03/28 09:19:07 WARN clients.NetworkClient: [Consu ...
window环境启动kafka报错：此时不应有../config/log4j.properties
window7下启动kafka报错如下: 背景: 1.在官网下载kafka_2.10-0.10.2.0.tgz: 2.将第一步的压缩包解压到D:\Program Files (x86)目录下: 3.启 ...
win10 启动kafka报错另一个程序正在使用此文件，进程无法访问
welcome to my blog 问题描述: win10启动kafka报错: ERROR Failed to clean up log for __consumer_offsets-39 in d ...
ueditor上传图片回调_（常见解决方法）UEditor报错“后端配置项没有正常加载，上传插件不能正常使用”...
(常见解决方法)UEditor报错"后端配置项没有正常加载,上传插件不能正常使用"_向来萧瑟也无畏-CSDN博客blog.csdn.net 报错信息详见此文的"排错过 ...
搭建dubbo项目解决dubbo.xml标签报错的问题
搭建dubbo项目解决dubbo.xml标签报错的问题参考文章: (1)搭建dubbo项目解决dubbo.xml标签报错的问题 (2)https://www.cnblogs.com/ajax-li/ ...

解决flume整合kafka报错Attempt to heart beat failed since member id is not valid, reset it and try to re-jo

报错情景:

原因:

解决办法:

解决flume整合kafka报错Attempt to heart beat failed since member id is not valid, reset it and try to re-jo相关推荐

最新文章

热门文章