HMaster因zookeeper连接超时自杀

缘由

hbase在上线一段时间后，发现HMaster经常会自杀挂掉，具体报错如下

报错

2019-06-14 11:24:07,242 WARN  [master/ms-fibo-test-dataserver4/172.16.201.239:16000-EventThread] client.ConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs it
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expiredat org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:634)at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:566)at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40)at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534)at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2019-06-14 11:24:07,243 INFO  [master/ms-fibo-test-dataserver4/172.16.201.239:16000-EventThread] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x26a9d2b8a210323
2019-06-14 11:24:07,243 INFO  [master/ms-fibo-test-dataserver4/172.16.201.239:16000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2019-06-14 11:24:07,466 INFO  [main-SendThread(ms-fibo-test-dataserver5:2181)] zookeeper.ClientCnxn: Opening socket connection to server ms-fibo-test-dataserver5/172.16.201.240:2181. Will not attempt to authenticate using SASL (unknown error)
2019-06-14 11:24:07,468 INFO  [main-SendThread(ms-fibo-test-dataserver5:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /172.16.201.239:55808, server: ms-fibo-test-dataserver5/172.16.201.240:2181
2019-06-14 11:24:07,479 INFO  [main-SendThread(ms-fibo-test-dataserver5:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x36a9d2b8896031f has expired, closing socket connection
2019-06-14 11:24:07,479 FATAL [main-EventThread] master.HMaster: master:16000-0x36a9d2b8896031f, quorum=ms-fibo-test-dataserver4.fibodata.com:2181,ms-fibo-test-dataserver5.fibodata.com:2181,ms-fibo-test-dataserver6.fibodata.com:2181, baseZNode=/hbase-unsecure master:16000-0x36a9d2b8896031f received expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expiredat org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:634)at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:566)at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534)

解决方案

由于网络抖动或者zookeeper full gc导致连接超时，会导致hbase连接不上zookeeper超时而挂掉。

1.调大hbase会话超时时间： hbase-site.xml

<property><name>zookeeper.session.timeout</name><value>240000</value><!--默认: 180000 ：zookeeper 会话超时时间，单位是毫秒 -->
</property>

2.调大zookeeper会话超时时间：zoo.cfg

# 默认3000毫米OA
tickTime=5000Session超时时间限制，如果客户端设置的超时时间不在这个范围，那么会被强制设置为最大或最小时间。默认的Session超时时间是在2 *  tickTime ~ 20 * tickTime 这个范围 New in 3.3.0

HMaster因zookeeper连接超时自杀相关推荐

hbase 查询设置超时_hbase master挂掉-zookeeper连接超时原因
并行运行hbase删表,建表操作,多个表多个region,导致hbase挂掉.查看日志: 从日志中可以看出GC时间过长导致zookeeper连接超时,master退出.(是master退出而不是reg ...
【Redis连接超时】io.lettuce.core.RedisConnectionException: Unable to connect to 192.168.x.x:7000
「扫码关注我,面试.各种技术(mysql.zookeeper.微服务.redis.jvm)持续更新中-」今天临近下班了,线上开始频繁报警,各种Redis连接超时,顿时慌的一批,因为最近在优化系统高频 ...
elasticjob任务启动zk连接超时问题解决
版本信息 jdk1.8 elasticjob 3.0.1 spring boot 1.5.22.RELEASE 异常信息 Caused by: org.apache.shardingsphere.el ...
mysql导入sql文件过大或连接超时的解决的方法
前段时间出差在现场开发的时候,导入数据库老是出问题.最后发现了一个奇妙sql语句交给实施,仅仅要导入出错就把例如以下语句运行就能够了.至今屡试不爽. set global max_allowed_pa ...
修正版 | QPS过万，Redis大量连接超时怎么解决？
之前负责的一个服务总是在高峰时刻和压测发生大量的redis连接超时的异常redis.clients.jedis.exceptions.JedisConnectionException,根据原有的业务规 ...
QPS过万，redis大量连接超时怎么解决？
7月2号10点后,刚好某个负责的服务发生大量的redis连接超时的异常(redis.clients.jedis.exceptions.JedisConnectionException),由于本身的数据 ...
服务器连接超时不响应怎么回事,勇者斗恶龙x服务器没有响应连接超时解决方法...
勇者斗恶龙x服务器没有响应连接超时怎么办?勇者斗恶龙x服务器没有响应连接超时解决方法.今天小编就带大家一起来看一下这一问题的解决方法,一起来看一下吧! (24日国服首测正式开启) 勇者斗恶龙10进不去 ...
如何解决MySQL连接超时关闭
最近做网站有一个站要用到WEB网页采集器功能,当一个PHP脚本在请求URL的时候,可能这个被请求的网页非常慢慢,超过了mysql的 wait-timeout时间,然后当网页内容被抓回来后,准备插入到M ...
SSRS报表连接超时的问题
这段时间遇到一个问题就是ReportService 中采用了远程连接的报表偶尔会断开连接,导致报表导出异常,查阅了很多资料,几天来就是断断续续的终于解决了这个问题,下面把一些解决的点一一展示出来,便于 ...

HMaster因zookeeper连接超时自杀

缘由

报错

解决方案

HMaster因zookeeper连接超时自杀相关推荐

最新文章

热门文章