应用程序连接hbase报错：java.net.SocketTimeoutException: callTimeout=60000

背景说明：

　　今天对生产环境hbase增加了节点，下午的时候一个同事反馈，应用程序后台报错，如下：

Tue Feb 26 17:35:35 CST 2019, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68451: row 'SYSTEM.CATALOG,TARGETCUST_DATA,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host-10-191-36-24,16020,1551146724629, seqNum=0at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276)at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:210)at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212)at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186)at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1275)at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165)at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1122)at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:957)at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:506)at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:722)at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:692)at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1769)at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1724)at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1704)at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1301)... 47 more
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68451: row 'SYSTEM.CATALOG,TARGETCUST_DATA,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host-10-191-36-24,16020,1551146724629, seqNum=0at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:169)at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)... 3 more
Caused by: java.net.UnknownHostException: host-10-191-36-24at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.<init>(AbstractRpcClient.java:315)at org.apache.hadoop.hbase.ipc.AbstractRpcClient.createBlockingRpcChannel(AbstractRpcClient.java:267)at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getClient(ConnectionManager.java:1639)at org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:162)at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.prepare(ScannerCallableWithReplicas.java:376)at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)... 4 more
2019-02-26 17:35:35 [com.asiainfo.cb2.consumer.ReceiveSMSID]-[ERROR]:33 - sms consumer begin
2019-02-26 17:35:35 [com.asiainfo.cb2.consumer.ReceiveSMSID]-[ERROR]:38 - sms consumer pushType3

分析：

　　开始的时候，思路还是纠结在，超时的错误java.net.SocketTimeoutException: callTimeout=60000,想着有没有办法能够增加客户端的超时时间，然后还看了datanode的日志，认为是datanode在写数据导磁盘的时候性能问题，新增加hadoop节点，节点间在进行平衡，导致影响了性能，但是，对于以上的分析都没有更好的方法来解决问题。

结果：

　　等到后续，在仔细看错误，发现，Caused by: java.net.UnknownHostException: host-10-191-36-24，找不到主机的异常，才突然明白，应用程序首先连接到zk，然后zk告知region在哪个regionserver上，然后，应用程序再连接到hbase的regionserver上读写数据。

解决：

　　所以，在应用程序的/etc/hosts上配置新增的hbase regionserver节点的hosts解析，再次观察应用程序，该问题解决

文档创建时间：2019年2月27日10:34:38

转载于:https://www.cnblogs.com/chuanzhang053/p/10442044.html

应用程序连接hbase报错：java.net.SocketTimeoutException: callTimeout=60000相关推荐

解决Hbase报错java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for....
完整报错为: java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper ...
java程序连接MinIO 报错The request signature we calculated does not match the signature you provided.
java程序连接MinIO 报错The request signature we calculated does not match the signature you provided. Check ...
Hbase报错解决方法ERROR: KeeperErrorCode = NoNode for /hbase/master
步骤今天搭建Hbase环境,遇到了此bug.一开始搭建hdfs的时候还是很顺利的,参考着别人的博客,一步一的做下去,最后可以正常启动.而在搭建HBase的时候遇到了一个很大的bug.但是造成该bug ...
DRP问题集结（一）-Tomcat无法启动,报错java.lang.NoClassDefFoundError: org/apache/juli/logging/LogFactory...
问题一: Tomcat无法启动,报错java.lang.NoClassDefFoundError: org/apache/juli/logging/LogFactory 问题二:[Error]Jav ...
WildFly 报错 java.lang.NoClassDefFoundError
在eclipse上WildFly部署项目后,启动一直报错java.lang.NoClassDefFoundError,功夫不负有心人,终于解决. 解决方案查了网上很多资料,有说环境变量配置不对的,有 ...
Eclipse启动报错Java was started but returned exit code=13
启动Eclipse的时候报错Java was started but returned exit code=13,这个错误的原因是由于eclipse版本与jdk版本不符导致的,可能你的eclipse是 ...
maven引用servlet_解决Maven 项目报错 java.httpservlet和synchronized使用方法
使用java8 的lanmbe表达式时,使用java1.8编译,则会报错需要在pom.xml的中添加 org.apache.maven.plugins maven-compiler-plugin 2 ...
python hbase 报错by_【hbase】使用thrift with python 访问HBase
HBase 版本: 0.98.6 thrift 版本: 0.9.0 使用 thrift client with python 连接 HBase 报错: 1 Traceback (most rece ...
首次使用eclipes运行项目报错“Java.lang.ClassNotFoundException”
首次使用eclipes运行项目报错"Java.lang.ClassNotFoundException" 运行项目时提示: 解决方法: 1.点击菜单-Project-Build Au ...

应用程序连接hbase报错：java.net.SocketTimeoutException: callTimeout=60000

应用程序连接hbase报错：java.net.SocketTimeoutException: callTimeout=60000相关推荐

最新文章

热门文章