Kafka服务不可用(宕机)问题踩坑记
背景
某线上日志收集服务报警,打开域名报502错误码。 收集服务由2台netty HA服务器组成,netty服务器将客户端投递来的protobuf日志解析并发送到kafka,打开其中一个应用的日志,发现如下报错:
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s)
在排除了netty服务的错误之后,去查看kafka的日志。 发现报错,排查过程如下;
配置信息
系统 | kafka版本 | broker数量 |
---|---|---|
CentOS7.4 | 2.1.0 | 3 |
线上有三台Kafka Broker,id分别为0、1、2,服务器只部署了Kafka服务。
问题
线程是否存活
首先jps查看Kafka线程是否存活,三台机器都没问题,kafka依然在运行。
GC问题
查看kafkaServer-gc.log.1.current的日志,gc日志没发现异常。
Broker 0/server.log
[2019-08-02 15:17:03,699] WARN Attempting to send response via channel for which there is no open connection, connection id 172.21.3.14:9092-172.21.3.11:54311-107706 (kafka.network.Processor)
[2019-08-02 15:19:12,490] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager) [2019-08-02 15:26:54,405] INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Error sending fetch request (sessionId=1112819217, epoch=1897450) to node 1: java.io.IOException: Connection to 1 was disconnected before the response was read. (org.apache.kafka.clients.FetchSessionHandler) [2019-08-02 15:26:54,411] WARN [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId=0, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={galaxy_client-7=(offset=15680912, logStartOffset=14755985, maxBytes=1048576, currentLeaderEpoch=Optional[9])}, isolationLevel=READ_UNCOMMITTED, toForget=, metadata=(sessionId=1112819217, epoch=1897450)) (kafka.server.ReplicaFetcherThread) java.io.IOException: Connection to 1 was disconnected before the response was read at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97) at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:97) at kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:190) at kafka.server.AbstractFetcherThread.kafka$server$AbstractFetcherThread$$processFetchRequest(AbstractFetcherThread.scala:241) at kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:130) at kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:129) at scala.Option.foreach(Option.scala:257) at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:111) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82) [2019-08-02 15:27:26,433] INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Error sending fetch request (sessionId=1112819217, epoch=INITIAL) to node 1: java.io.IOException: Connection to 1 was disconnected before the response was read. (org.apache.kafka.clients.FetchSessionHandler)
Broker 1/server.log
[2019-08-02 15:26:53,751] WARN [GroupCoordinator 1]: Failed to write empty metadata for group StoreToHiveV2: The group is rebalancing, so a rejoin is needed. (kafka.coordinator.group.GroupCoordinator)
[2019-08-02 15:26:55,515] WARN [GroupCoordinator 1]: Failed to write empty metadata for group LPmkt: The group is rebalancing, so a rejoin is needed. (kafka.coordinator.group.GroupCoordinator) [2019-08-02 15:26:56,124] WARN [GroupCoordinator 1]: Failed to write empty metadata for group Store2CarbonClientServer: The group is rebalancing, so a rejoin is needed. (kafka.coordinator.group.GroupCoordinator) [2019-08-02 15:26:56,575] WARN [GroupCoordinator 1]: Failed to write empty metadata for group StoreToHive: The group is rebalancing, so a rejoin is needed. (kafka.coordinator.group.GroupCoordinator)
Broker 2/server.log
[2019-08-02 15:26:54,514] WARN [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId=2, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={galaxy_client-7=(offset=15680912, logStartOffset=14755985, maxBytes=1048576, currentLeaderEpoch=Optional[9])}, isolationLevel=READ_UNCOMMITTED, toForget=, metadata=(sessionId=1550207796, epoch=1890003)) (kafka.server.ReplicaFetcherThread) java.io.IOException: Connection to 1 was disconnected before the response was read at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97) at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:97) at kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:190) at kafka.server.AbstractFetcherThread.kafka$server$AbstractFetcherThread$$processFetchRequest(AbstractFetcherThread.scala:241) at kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:130) at kafka.server.AbstractFetcherThread$$anonfun$maybeFetch$1.apply(AbstractFetcherThread.scala:129) at scala.Option.foreach(Option.scala:257) at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:111) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82) [2019-08-02 15:27:26,569] INFO [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Error sending fetch request (sessionId=1550207796, epoch=INITIAL) to node 1: java.io.IOException: Connection to 1 was disconnected before the response was read. (org.apache.kafka.clients.FetchSessionHandler)
问题排查
由于是线上应用,基本没时间排查问题,所以立即重启了三台Kafka,重启后得到缓解。
在服务正常之后,在网上搜了很多答案,有的人说是网络问题,close wait过多导致broker网络不通,从而Kafka产生脑裂问题。 在各个服务器上运行命令:
netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
结果
ESTABLISHED 172
TIME_WAIT 32
也没发现异常,等下次再出现问题时,要运行这个命令看看服务器的状态。
社区bug?
https://issues.apache.org/jira/browse/KAFKA-6582
下方有人评论:We are running 2.1.1 in production (and four other environments) since March without this issue showing again.
所以找时间升级Kafka版本吧~
转载于:https://www.cnblogs.com/ChouYarn/p/11290396.html
Kafka服务不可用(宕机)问题踩坑记相关推荐
- go nil json.marshal 完是null_字节跳动踩坑记#3:Go服务灵异panic
这个坑比较新鲜,刚填完,还冒着冷气. - 1 - 在字节跳动,我们服务的所有 log 都通过统一的日志库采集到流式日志服务.落地 ES 集群,配上字节云超(sang)级(xin)强(bing)大(ku ...
- 蓝鲸社区版单机、多机部署踩坑分享
蓝鲸社区版单机.多机部署踩坑分享 一.单机版部署 1.报错:current host is not the controller 解决方法: cat /data/install/.controller ...
- 在 Debian 上将 aria2c 封装为服务的方法及遇到的踩坑点
在 Debian 上将 aria2c 封装为服务的方法及遇到的踩坑点 问题起因 Aria2c 的用途就不介绍了,由于 aria2c 并没有提供服务封装,这就导致了如果需要启动.停止和重启时,都必须手工 ...
- spug上线服务踩坑记
spug是一款优秀的自动化运维平台, 这让我们想自动化又向前迈了一步. 在过程中遇到的问题,记录在这里供各位看官参考. 目录 一.远程执行命令问题 二.软连接的问题 三.jdk 版本问题 四.ssh ...
- 服务器重新部署踩坑记
服务器重新部署踩坑记 Intro 之前的服务器是 Ubuntu 18.04 ,上周周末想升级一下服务器系统,从 18.04 升级到 20.04,结果升级升挂了... 后来 SSH 始终连不上,索性删除 ...
- Typora+PicGo-core+SMMS图床踩坑记
Typora+PicGo-core+SMMS图床踩坑记 一.安装PicGo-Core 二.安装smms v2 API插件 三.配置PicGo-Core 四.最终效果 五.总结 最近把Typora更新后 ...
- 7代cpu能装虚拟xp系统吗_Intel 10代PC/笔记本安装Win7踩坑记 amp; 驱动分享
免责声明:本文只是从技术角度进行讨论,主要因我自己的兴趣而引发测试.只代表个人观点,与任何组织机构无关. 目录 - 从赛扬G5900"亮机"CPU说起 - 10代Core Wi ...
- python从入门到实践django看不懂_Python编程:从入门到实践踩坑记 Django
<>踩坑记 Django Django Python 19.1.1.5 模板new_topic 做完书上的步骤后,对主题添加页面经行测试,但是浏览器显示 服务器异常. 个人采用的开发环境是 ...
- 东八区转为0时区_踩坑记 | Flink 天级别窗口中存在的时区问题
❝ 本系列每篇文章都是从一些实际的 case 出发,分析一些生产环境中经常会遇到的问题,抛砖引玉,以帮助小伙伴们解决一些实际问题.本文介绍 Flink 时间以及时区问题,分析了在天级别的窗口时会遇到的 ...
最新文章
- 提取图像数据的特征,让机器“看见”
- android evaluater_android通过WebView的evaluateJavascript()调用JS
- std:forward 完美转发
- Qt Creator使用自定义字体
- 解决嵌入式开发过程中开发板与虚拟机的网络通信问题
- python 知识 rstrip,strip,lstrip
- 机器学习(part4)--模型评估
- Java基础 深拷贝浅拷贝
- MySQL-->索引-->如何创建索引,创建原则
- 如何优雅的统计代码耗时?
- 数据库服务器信息填写,数据库服务器是怎么填写
- 微信测试账户申请以及测试
- linux系统认证中级是什么,红帽中级RHCE证书有什么用
- 概率统计_假设检验——单边检验的接受域和拒绝域
- python 处理pdf文件 转成txt 批量提取pdf中的文字
- [jzoj 4249] 【五校联考7day1】游戏 {贪心/斜率优化}
- ubuntu linux下直观的网络流量监控
- win10环境下配置Gradle
- 一个屌丝程序猿的人生(二十四)
- 计算机网络-15 网络测量