2013/08/09 转发自http://bkeep.blog.163.com/blog/static/123414290201272644422987/

【案例】dfs.datanode.max.xcievers参数导致hbase-0.92集群报错

2012-08-26 16:44:22|  分类： Hbase|字号 订阅

场景:

15个datanode挂掉，只有2个存活

[dwhftp@dw-hbase-1 ~]$ hadoop dfsadmin -report

Configured Capacity: 73837983129600 (67.16 TB)

Present Capacity: 69740285348454 (63.43 TB)

DFS Remaining: 61837580668928 (56.24 TB)

DFS Used: 7902704679526 (7.19 TB)

DFS Used%: 11.33%

Under replicated blocks: 0

Blocks with corrupt replicas: 0

Missing blocks: 0

14个regionserver只有2个存活

[dwhftp@dw-hbase-11 ~]$ hbase shell

HBase Shell; enter 'help<RETURN>' for list of supported commands.

Type "exit<RETURN>" to leave the HBase Shell

Version 0.92.0, r1231986, Mon Jan 16 13:16:35 UTC 2012

hbase(main):001:0> status

14 servers, 0 dead, 4739.8571 average load

服务进程没有挂掉，但不能对外提供服务，且服务器内存消耗16G

dwhftp   29754  1 15 Jul30 ?        3-07:58:27 /usr/alibaba/java/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx16000m -Dhbase.log.dir=/home/dwhftp/opt/hbase/logs -Dhbase.log.file=hbase-dwhftp-regionserver-dw-hbase-9.hst.ali.dw.alidc.net.log -Dhbase.home.dir=/home/dwhftp/opt/hbase -Dhbase.id.str=dwhftp -Dhbase.root.logger=INFO,DRFA -Djava.library.path=/home/dwhftp/opt/hbase/lib/native/Linux-amd64-64 -classpath /home/dwhftp/opt/hbase/conf:/usr/alibaba/java/lib/tools.jar:/home/dwhftp/opt/hbase:/home/dwhftp/opt/hbase/hbase-0.92.0.jar:/home/dwhftp/opt/hbase/hbase-0.92.0-tests.jar:/home/dwhftp/opt/hbase/lib/activation-1.1.jar:/home/dwhftp/opt/hbase/lib/asm-3.1.jar:/home/dwhftp/opt/hbase/lib/avro-1.5.3.jar:/home/dwhftp/opt/hbase/lib/avro-ipc-1.5.3.jar:/home/dwhftp/opt/hbase/lib/commons-beanutils-1.7.0.jar:/home/dwhftp/opt/hbase/lib/commons-beanutils-core-1.8.0.jar:/home/dwhftp/opt/hbase/lib/commons-cli-1.2.jar:/home/dwhftp/opt/hbase/lib/commons-codec-1.4.jar:/home/dwhftp/opt/hbase/lib/commons-collections-3.2.1.jar:/home/dwhftp/opt/hbase/lib/commons-configuration-1.6.jar:/home/dwhftp/opt/hbase/lib/commons-digester-1.8.jar:/home/dwhftp/opt/hbase/lib/commons-el-1.0.jar:/home/dwhftp/opt/hbase/lib/commons-httpclient-3.1.jar:/home/dwhftp/opt/hbase/lib/commons-lang-2.5.jar:/home/dwhftp/opt/hbase/lib/commons-logging-1.1.1.jar:/home/dwhftp/opt/hbase/lib/commons-math-2.1.jar:/home/dwhftp/opt/hbase/lib/commons-net-1.4.1.jar:/home/dwhftp/opt/hbase/lib/core-3.1.1.jar:/home/dwhftp/opt/hbase/lib/guava-r09.jar:/home/dwhftp/opt/hbase/lib/guava-r09-jarjar.jar:/home/dwhftp/opt/hbase/lib/hadoop-core-0.20.2-cdh3u3.jar:/home/dwhftp/opt/hbase/lib/high-scale-lib-1.1.1.jar:/home/dwhftp/opt/hbase/lib/httpclient-4.0.1.jar:/home/dwhftp/opt/hbase/lib/httpcore-4.0.1.jar:/home/dwhftp/opt/hbase/lib/jackson-core-asl-1.5.5.jar:/home/dwhftp/opt/hbase/lib/jackson-jaxrs-1.5.5.jar:/home/dwhftp/opt/hbase/lib/jackson-mapper-asl-1.5.5.jar:/home/dwhftp/opt/hbase/lib/jackson-xc-1.5.5.jar:/home/dwhftp/opt/hbase/lib/jamon-runtime-2.3.1.jar:/home/dwhftp/opt/hbase/lib/jasper-compiler-5.5.23.jar:/home/dwhftp/opt/hbase/lib/jasper-runtime-5.5.23.jar:/home/dwhftp/opt/hbase/lib/jaxb-api-2.1.jar:/home/dwhftp/opt/hbase/lib/jaxb-impl-2.1.12.jar:/home/dwhftp/opt/hbase/lib/jersey-core-1.4.jar:/home/dwhftp/opt/hbase/lib/jersey-json-1.4.jar:/home/dwhftp/opt/hbase/lib/jersey-server-1.4.jar:/home/dwhftp/opt/hbase/lib/jettison-1.1.jar:/home/dwhftp/opt/hbase/lib/jetty-6.1.26.jar:/home/dwhftp/opt/hbase/lib/jetty-util-6.1.26.jar:/home/dwhftp/opt/hbase/lib/jruby-complete-1.6.5.jar:/home/dwhftp/opt/hbase/lib/jsp-2.1-6.1.14.jar:/home/dwhftp/opt/hbase/lib/jsp-api-2.1-6.1.14.jar:/home/dwhftp/opt/hbase/lib/libthrift-0.7.0.jar:/home/dwhftp/opt/hbase/lib/log4j-1.2.16.jar:/home/dwhftp/opt/hbase/lib/netty-3.2.4.Final.jar:/home/dwhftp/opt/hbase/lib/protobuf-java-2.4.0a.jar:/home/dwhftp/opt/hbase/lib/servlet-api-2.5-6.1.14.jar:/home/dwhftp/opt/hbase/lib/servlet-api-2.5.jar:/home/dwhftp/opt/hbase/lib/slf4j-api-1.5.8.jar:/home/dwhftp/opt/hbase/lib/slf4j-log4j12-1.5.8.jar:/home/dwhftp/opt/hbase/lib/snappy-java-1.0.3.2.jar:/home/dwhftp/opt/hbase/lib/stax-api-1.0.1.jar:/home/dwhftp/opt/hbase/lib/velocity-1.7.jar:/home/dwhftp/opt/hbase/lib/xmlenc-0.52.jar:/home/dwhftp/opt/hbase/lib/zookeeper-3.4.2.jar::/home/dwhftp/opt/hadoop/conf:/home/dwhftp/opt/hadoop/conf org.apache.hadoop.hbase.regionserver.HRegionServer start

hbase查看表时报错如下

hbase(main):001:0> scan '20120819entry'

ROW                           COLUMN+CELL

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/home/dwhftp/opt/install/hbase-0.92.0/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/home/dwhftp/opt/install/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

ERROR: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1

at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2819)

at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1755)

at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)

at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)

regionserver报错日志

org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/.logs/dw-hbase-17,60020,1345529427507/dw-hbase-17%2C60020%2C1345529427507.1345529431034 File does not exist. [Lease.  Holder: DFSClient_-1871695140, pendingcreates: 4]

省略部分文字...

2012-08-21 14:35:13,530 WARN org.apache.hadoop.hdfs.DFSClient: Error while syncing

java.io.IOException: Bad connect ack with firstBadLink as 172.16.197.18:50010

HMaster 上的报错

[dwhftp@dw-hbase-3 logs]$ pwd

/home/dwhftp/opt/hbase/logs

[dwhftp@dw-hbase-3 logs]$ tail -200f hbase-dwhftp-master-dw-hbase-3.hst.ali.dw.alidc.net.log

2012-08-21 10:37:56,548 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of .META.,,1 at address=dw-hbase-9,60020,1343643389530; org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1

hadoop日志（问题终于找到了！）

[dwhftp@dw-hbase-13 logs]$ pwd

/home/dwhftp/opt/hadoop/logs

[dwhftp@dw-hbase-13 logs]$ vi hadoop-dwhftp-datanode-dw-hbase-13.hst.ali.dw.alidc.net.log

2012-08-21 14:35:00,203 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.16.197.25:50010, storageID=DS-2042732685-172.16.197.25-50010-1334122560477, infoPort=50075, ipcPort=50020):DataXceiver

java.io.IOException: xceiverCount 4097 exceeds the limit of concurrent xcievers 4096

at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:156)

解决办法，调整xcievers参数

默认是4096，改为8192

vi /home/dwhftp/opt/hadoop/conf/hdfs-site.xml

<property>

<name>dfs.datanode.max.xcievers</name>

<value>8192</value>

</property>

dfs.datanode.max.xcievers 参数说明

一个 Hadoop HDFS Datanode 有一个同时处理文件的上限. 这个参数叫 xcievers (Hadoop的作者把这个单词拼错了). 在你加载之前，先确认下你有没有配置这个文件conf/hdfs-site.xml里面的xceivers参数，至少要有4096:

<property>

<name>dfs.datanode.max.xcievers</name>

<value>4096</value>

</property>

对于HDFS修改配置要记得重启.

如果没有这一项配置，你可能会遇到奇怪的失败。你会在Datanode的日志中看到xcievers exceeded，但是运行起来会报missing blocks错误。例如: 10/12/08 20:10:31 INFO hdfs.DFSClient: Could not obtain block blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node: java.io.IOException: No live nodes contain current block. Will get new block locations from namenode and retry... [5]

重启hbase和hdfs

hbase-1

[dwhftp@dw-hbase-1 ~]$ start-dfs.sh

hbase-3

[dwhftp@dw-hbase-3 ~]$ start-hbase.sh

hbase-3

还要启动mapper/reduce用于和云梯同步数据

[dwhftp@dw-hbase-3 ~]$start-mapred.sh

后续动作

添加监控

端口监控、进程监控、

hbase-1 (namenode) 9000  50070  80  2181  3555

hbase-2 (SecondaryNameNode) 50090  2181  3555

hbase-3 (HMaster) 60010  80  2181  3555

hbase-4 (HMaster备) 60000  2181  3555

hbase-5 （HRegionServer） 60020  60030  2181  3555  datanode  50010  50075

hbase-6 ~ dw-hbase-18 （HRegionServer） 60020  60030  datanode  50010  50075

自己写脚本监控

监控活动的hbase和datanode

监控日志中的ERROR关键字

容量

跟开发人员约定可用性

namenode是单点，所以肯定不是7*24小时服务。单台机器挂掉hbase集群需要时间重建hbase，分钟级别的不可用。必须跟他们讲清楚。然后确定谁在维护hbase

另外创建一个hbase维护群，方便大家交流。

hbase资源申请

所以应用共同使用hbase集群，所以资源隔离。业务评审都需要有流程和规范。

日志级别调整

info、debug、warn、error放在同一个日志文件里面，信息量太多导致排错时间长

通过配置log4j将error级别日志打到单独文件里面。

向淘宝取经

日常维护动作

监控点