==版本信息==

HBase：2.7.1

Storm：1.0.1

RocketMQ：3.4.6（阿里版）

==问题描述==

2018年9月3号晚上23点左右，例行巡检系统运行状况时，

发现Storm消费RocketMQ入HBase的过程中，RocketMQ的TPS极低，甚至为0了。

（Storm从RocketMQ消费消息自后，读写HBase）

==问题定位==

查看了Storm的日志，发现有如下异常信息：

2018-09-03 20:29:43.696 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] java.net.SocketTimeoutException: callTimeout=1200000, callDuration=1536845: row '97390002332830' on table 'DBN_ANE' at region=DBN_ANE,97,1507795766296.69724c5bd4e82bae4a3ec8a95c8f9c65., hostname=dscn33,16020,1529215379398, seqNum=17296637
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.HTable.get(HTable.java:862)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.HTable.get(HTable.java:828)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at com.dscn.topology.BasicBolt.execute(BasicBolt.java:125)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:50)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.daemon.executor$fn__5030$tuple_action_fn__5032.invoke(executor.clj:729)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.daemon.executor$mk_task_receiver$fn__4951.invoke(executor.clj:461)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.disruptor$clojure_handler$reify__4465.onEvent(disruptor.clj:40)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:482)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:460)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.daemon.executor$fn__5030$fn__5043$fn__5096.invoke(executor.clj:848)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at clojure.lang.AFn.run(AFn.java:22)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at java.lang.Thread.run(Thread.java:745)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] Caused by: java.io.IOException: HRegionInfo was null in DBN_ANE, row=keyvalues={DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf./info:seqnumDuringOpen/1535968873207/Put/vlen=8/seqid=0, DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf./info:server/1535968873207/Put/vlen=12/seqid=0, DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf./info:serverstartcode/1535968873207/Put/vlen=8/seqid=0}
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1289)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1152)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1136)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:957)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:75)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] ... 14 more

然后查看HBase集群日志，里面也出现错误，日志如下：

2018-09-03 17:58:57,603 WARN  [MemStoreFlusher.0] regionserver.HStore: Failed flushing store file, retrying num=0
java.io.IOException: All datanodes 10.11.2.31:50010 are bad. Aborting...at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137)at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
2018-09-03 17:58:57,606 INFO  [regionserver/dscn33/10.11.2.31:16020.logRoller] wal.FSHLog: Slow sync cost: 16261 ms, current pipeline: [10.11.2.31:50010]
2018-09-03 17:58:57,606 ERROR [regionserver/dscn33/10.11.2.31:16020.logRoller] wal.FSHLog: Failed close of WAL writer hdfs://ns/hbase/WALs/dscn33,16020,1529215379398/dscn33%2C16020%2C1529215379398.default.1535968646203, unflushedEntries=30
org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=41639867, requesting roll of WALat org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1615)at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:833)at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:699)at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=41639867, requesting roll of WALat org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1898)at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1750)at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1672)at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)... 1 more
Caused by: java.io.IOException: All datanodes 10.11.2.31:50010 are bad. Aborting...at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137)at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2018-09-03 17:58:41,358 ERROR [MemStoreFlusher.1] regionserver.MemStoreFlusher: Cache flush failed for region DBN_JBD,30270769371,1515408107652.fca3d8ee774898302a377535e3f0ccac.
org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=41639867, requesting roll of WALat org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1898)at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1750)at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1672)at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: All datanodes 10.11.2.31:50010 are bad. Aborting...at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137)at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

检查HBase集群状态（hbase hbck），发现异常，定位为HBase集群问题：

[root@dscn12 ~]# hbase hbck
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadmin/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadmin/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBaseFsck command line options:
2018-09-04 00:01:54,217 INFO  [main] util.HBaseFsck: Launching hbck
2018-09-04 00:01:54,456 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x57576994 connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:01:54,471 INFO  [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2018-09-04 00:01:54,471 INFO  [main] zookeeper.ZooKeeper: Client environment:host.name=dscn12
2018-09-04 00:01:54,471 INFO  [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_73
2018-09-04 00:01:54,472 INFO  [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2018-09-04 00:01:54,472 INFO  [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_73/jre
2018-09-04 00:01:54,472 INFO  [main] zookeeper.ZooKeeper: Client environment:java.class.path=/home/hadmin/hbase/conf:/usr/java/default/lib/tools.jar:/home/hadmin/hbase:/home/hadmin/hbase/lib/activation-1.1.jar:/home/hadmin/hbase/lib/antisamy-1.4.3.jar:/home/hadmin/hbase/lib/aopalliance-1.0.jar:/home/hadmin/hbase/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadmin/hbase/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadmin/hbase/lib/api-asn1-api-1.0.0-M20.jar:/home/hadmin/hbase/lib/api-util-1.0.0-M20.jar:/home/hadmin/hbase/lib/asm-3.1.jar:/home/hadmin/hbase/lib/avro-1.7.4.jar:/home/hadmin/hbase/lib/batik-css-1.7.jar:/home/hadmin/hbase/lib/batik-ext-1.7.jar:/home/hadmin/hbase/lib/batik-util-1.7.jar:/home/hadmin/hbase/lib/bsh-core-2.0b4.jar:/home/hadmin/hbase/lib/commons-beanutils-1.7.0.jar:/home/hadmin/hbase/lib/commons-beanutils-core-1.7.0.jar:/home/hadmin/hbase/lib/commons-cli-1.2.jar:/home/hadmin/hbase/lib/commons-codec-1.9.jar:/home/hadmin/hbase/lib/commons-collections-3.2.2.jar:/home/hadmin/hbase/lib/commons-compress-1.4.1.jar:/home/hadmin/hbase/lib/commons-configuration-1.6.jar:/home/hadmin/hbase/lib/commons-daemon-1.0.13.jar:/home/hadmin/hbase/lib/commons-digester-1.8.jar:/home/hadmin/hbase/lib/commons-el-1.0.jar:/home/hadmin/hbase/lib/commons-fileupload-1.2.jar:/home/hadmin/hbase/lib/commons-httpclient-3.1.jar:/home/hadmin/hbase/lib/commons-io-2.4.jar:/home/hadmin/hbase/lib/commons-lang-2.6.jar:/home/hadmin/hbase/lib/commons-logging-1.2.jar:/home/hadmin/hbase/lib/commons-math-2.2.jar:/home/hadmin/hbase/lib/commons-math3-3.1.1.jar:/home/hadmin/hbase/lib/commons-net-3.1.jar:/home/hadmin/hbase/lib/disruptor-3.3.0.jar:/home/hadmin/hbase/lib/esapi-2.1.0.jar:/home/hadmin/hbase/lib/findbugs-annotations-1.3.9-1.jar:/home/hadmin/hbase/lib/guava-12.0.1.jar:/home/hadmin/hbase/lib/guice-3.0.jar:/home/hadmin/hbase/lib/guice-servlet-3.0.jar:/home/hadmin/hbase/lib/hadoop-annotations-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-auth-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-client-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-common-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-hdfs-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-app-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-common-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-core-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-api-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-client-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-common-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-server-common-2.5.1.jar:/home/hadmin/hbase/lib/hbase-annotations-1.2.1.jar:/home/hadmin/hbase/lib/hbase-annotations-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-client-1.2.1.jar:/home/hadmin/hbase/lib/hbase-common-1.2.1.jar:/home/hadmin/hbase/lib/hbase-common-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-examples-1.2.1.jar:/home/hadmin/hbase/lib/hbase-external-blockcache-1.2.1.jar:/home/hadmin/hbase/lib/hbase-hadoop2-compat-1.2.1.jar:/home/hadmin/hbase/lib/hbase-hadoop-compat-1.2.1.jar:/home/hadmin/hbase/lib/hbase-it-1.2.1.jar:/home/hadmin/hbase/lib/hbase-it-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-prefix-tree-1.2.1.jar:/home/hadmin/hbase/lib/hbase-procedure-1.2.1.jar:/home/hadmin/hbase/lib/hbase-protocol-1.2.1.jar:/home/hadmin/hbase/lib/hbase-resource-bundle-1.2.1.jar:/home/hadmin/hbase/lib/hbase-rest-1.2.1.jar:/home/hadmin/hbase/lib/hbase-server-1.2.1.jar:/home/hadmin/hbase/lib/hbase-server-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-shell-1.2.1.jar:/home/hadmin/hbase/lib/hbase-thrift-1.2.1.jar:/home/hadmin/hbase/lib/htrace-core-3.1.0-incubating.jar:/home/hadmin/hbase/lib/httpclient-4.2.5.jar:/home/hadmin/hbase/lib/httpcore-4.4.1.jar:/home/hadmin/hbase/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hbase/lib/jackson-jaxrs-1.9.13.jar:/home/hadmin/hbase/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hbase/lib/jackson-xc-1.9.13.jar:/home/hadmin/hbase/lib/jamon-runtime-2.4.1.jar:/home/hadmin/hbase/lib/jasper-compiler-5.5.23.jar:/home/hadmin/hbase/lib/jasper-runtime-5.5.23.jar:/home/hadmin/hbase/lib/javax.inject-1.jar:/home/hadmin/hbase/lib/java-xmlbuilder-0.4.jar:/home/hadmin/hbase/lib/jaxb-api-2.2.2.jar:/home/hadmin/hbase/lib/jaxb-impl-2.2.3-1.jar:/home/hadmin/hbase/lib/jcodings-1.0.8.jar:/home/hadmin/hbase/lib/jersey-client-1.9.jar:/home/hadmin/hbase/lib/jersey-core-1.9.jar:/home/hadmin/hbase/lib/jersey-guice-1.9.jar:/home/hadmin/hbase/lib/jersey-json-1.9.jar:/home/hadmin/hbase/lib/jersey-server-1.9.jar:/home/hadmin/hbase/lib/jets3t-0.9.0.jar:/home/hadmin/hbase/lib/jettison-1.3.3.jar:/home/hadmin/hbase/lib/jetty-6.1.26.jar:/home/hadmin/hbase/lib/jetty-sslengine-6.1.26.jar:/home/hadmin/hbase/lib/jetty-util-6.1.26.jar:/home/hadmin/hbase/lib/joni-2.1.2.jar:/home/hadmin/hbase/lib/jruby-complete-1.6.8.jar:/home/hadmin/hbase/lib/jsch-0.1.42.jar:/home/hadmin/hbase/lib/jsp-2.1-6.1.14.jar:/home/hadmin/hbase/lib/jsp-api-2.1-6.1.14.jar:/home/hadmin/hbase/lib/jsr305-1.3.9.jar:/home/hadmin/hbase/lib/junit-4.12.jar:/home/hadmin/hbase/lib/leveldbjni-all-1.8.jar:/home/hadmin/hbase/lib/libthrift-0.9.3.jar:/home/hadmin/hbase/lib/log4j-1.2.17.jar:/home/hadmin/hbase/lib/metrics-core-2.2.0.jar:/home/hadmin/hbase/lib/nekohtml-1.9.12.jar:/home/hadmin/hbase/lib/netty-all-4.0.23.Final.jar:/home/hadmin/hbase/lib/paranamer-2.3.jar:/home/hadmin/hbase/lib/protobuf-java-2.5.0.jar:/home/hadmin/hbase/lib/servlet-api-2.5-6.1.14.jar:/home/hadmin/hbase/lib/servlet-api-2.5.jar:/home/hadmin/hbase/lib/slf4j-api-1.7.7.jar:/home/hadmin/hbase/lib/slf4j-log4j12-1.7.5.jar:/home/hadmin/hbase/lib/snappy-java-1.0.4.1.jar:/home/hadmin/hbase/lib/spymemcached-2.11.6.jar:/home/hadmin/hbase/lib/xalan-2.7.0.jar:/home/hadmin/hbase/lib/xml-apis-1.3.03.jar:/home/hadmin/hbase/lib/xml-apis-ext-1.3.04.jar:/home/hadmin/hbase/lib/xmlenc-0.52.jar:/home/hadmin/hbase/lib/xom-1.2.5.jar:/home/hadmin/hbase/lib/xz-1.0.jar:/home/hadmin/hbase/lib/zookeeper-3.4.6.jar:/home/hadmin/hadoop/etc/hadoop:/home/hadmin/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadmin/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/activation-1.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadmin/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/hadmin/hadoop/share/hadoop/common/lib/hadoop-auth-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/hadoop-annotations-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadmin/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hadmin/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hadmin/hadoop/share/hadoop/common/lib/xz-1.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadmin/hadoop/share/hadoop/common/lib/junit-4.11.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/home/hadmin/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadmin/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadmin/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/hadoop-common-2.7.1-tests.jar:/home/hadmin/hadoop/share/hadoop/common/hadoop-nfs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/hdfs:/home/hadmin/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadmin/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.1-tests.jar:/home/hadmin/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.1-tests.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.1.jar:.::/home/hadmin/hadoop/contrib/capacity-scheduler/*.jar:/home/hadmin/hadoop/bin
2018-09-04 00:01:54,472 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=/home/hadmin/hadoop/lib/native
2018-09-04 00:01:54,472 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2018-09-04 00:01:54,472 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2018-09-04 00:01:54,472 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2018-09-04 00:01:54,473 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2018-09-04 00:01:54,473 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-229.el7.x86_64
2018-09-04 00:01:54,473 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=root
2018-09-04 00:01:54,473 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/root
2018-09-04 00:01:54,473 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/root
2018-09-04 00:01:54,474 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hconnection-0x575769940x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:01:54,499 INFO  [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:01:54,506 INFO  [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
2018-09-04 00:01:54,564 INFO  [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bd9a, negotiated timeout = 40000
Version: 1.2.1
Number of live region servers: 25
Number of dead region servers: 0
Master: dscn11,16000,1529215352324
Number of backup masters: 1
Average load: 305.88
Number of requests: 2507
Number of regions: 7647
Number of regions in transition: 0
2018-09-04 00:01:56,473 INFO  [main] util.HBaseFsck: Loading regionsinfo from the hbase:meta table
ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
ERROR: Empty REGIONINFO_QUALIFIER found in hbase:metaNumber of empty REGIONINFO_QUALIFIER rows in hbase:meta: 2
2018-09-04 00:01:57,276 INFO  [main] util.HBaseFsck: getHTableDescriptors == tableNames => [DBN_BSHTIndex, DBN_TTKDIndex, SJYB_UPS, SJYB_ANE, DBN_SUR, DBN_CAEIndex, SJYB_BSHT, DBN_STOIndex, DBN_YTO, DBN_TTKD, DBN_FEDEX, SJYB_ZJS, DBN_ANEIndex, DBN_TNTIndex, SJYB_TNT, SJYB_DHL, DBN_ZJS, DBN_KJIndex, BOX, DBN_DEPPONIndex, DBN_ZGYZIndex, DBN_YUNDA, hbase:namespace, DBN_TNT, DBN_FEDEXIndex, DBN_YTOIndex, DBN_PJIndex, SJYB_EMS, SJYB_KJ, SJYB_ZGYZ, DBN_JBD, DBN_DEPPON, SJYB_KYE, SJYB_SUR, SJYB_PJ, SJYB_UC, DBN_SURIndex, DBN_EMS, DBN_STO, SJYB_TTKD, SJYB_SF, SJYB_JBD, SJYB_APEX, DBN_ZTOIndex, DBN_APEX, DBN_PJ, DBN_UPS, DBN_ZJSIndex, DBN_UPSIndex, DBN_UC, DBN_GTO, DBN_BSHT, DBN_EMSIndex, DBN_APEXIndex, SJYB_CAE, DBN_ZTO, DBN_CAE, DBN_SF, DBN_UCIndex, DBN_SFIndex, SJYB_YUNDA, SJYB_FEDEX, SJYB_STO, DBN_JBDIndex, DBN_ANE, DBN_DHLIndex, DBN_KYE, DBN_KJ, SJYB_GTO, DBN_ZGYZ, SJYB_YTO, SJYB_DEPPON, DBN_YUNDAIndex, SJYB_ZTO, DBN_DHL]
2018-09-04 00:01:57,280 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x51e8e6e6 connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:01:57,280 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hconnection-0x51e8e6e60x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:01:57,281 INFO  [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:01:57,281 INFO  [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
2018-09-04 00:01:57,283 INFO  [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bd9c, negotiated timeout = 40000
2018-09-04 00:01:57,296 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
2018-09-04 00:01:57,296 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2656ba391b6bd9c
2018-09-04 00:01:57,297 INFO  [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bd9c closed
2018-09-04 00:01:57,298 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
Number of Tables: 75
2018-09-04 00:01:57,308 INFO  [main] util.HBaseFsck: Loading region directories from HDFS
..............................................................................................................................................................................................................................................................................................................
2018-09-04 00:01:59,904 INFO  [main] util.HBaseFsck: Loading region information from HDFS
............................................................................
2018-09-04 00:02:02,282 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x7792d851 connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:02:02,282 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hconnection-0x7792d8510x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:02:02,286 INFO  [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:02:02,287 INFO  [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
2018-09-04 00:02:02,289 INFO  [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bda1, negotiated timeout = 40000
2018-09-04 00:02:02,303 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:02:02,303 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:02:02,304 INFO  [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:02:02,305 INFO  [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
2018-09-04 00:02:02,306 INFO  [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bda2, negotiated timeout = 40000
2018-09-04 00:02:02,352 INFO  [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bda2 closed
2018-09-04 00:02:02,352 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2018-09-04 00:02:02,352 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2656ba391b6bda1
2018-09-04 00:02:02,354 INFO  [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bda1 closed
2018-09-04 00:02:02,354 INFO  [main] util.HBaseFsck: Checking and fixing region consistency
2018-09-04 00:02:02,354 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
ERROR: Region { meta => null, hdfs => hdfs://ns/hbase/data/default/DBN_ANE/95a27c1ae2df5db14fa66d13966a8daf, deployed => dscn33,16020,1535968743109;DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf., replicaId => 0 } not in META, but deployed on dscn33,16020,1535968743109
ERROR: Region { meta => null, hdfs => hdfs://ns/hbase/data/default/DBN_ANE/e051ccd37678ff95fdd7652106b1fd5b, deployed => dscn33,16020,1535968743109;DBN_ANE,97,1535968864242.e051ccd37678ff95fdd7652106b1fd5b., replicaId => 0 } not in META, but deployed on dscn33,16020,1535968743109
2018-09-04 00:02:02,733 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: There is a hole in the region chain between 97 and 97581111303371.  You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table DBN_ANE
2018-09-04 00:02:02,876 INFO  [main] util.HBaseFsck: Computing mapping of all store files
...................................................................................................................................................................................................................................................................................................................................................................................
2018-09-04 00:02:10,477 INFO  [main] util.HBaseFsck: Validating mapping using HDFS state
2018-09-04 00:02:10,662 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:02:10,662 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:02:10,664 INFO  [main-SendThread(dscn11:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn11/10.11.2.4:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:02:10,665 INFO  [main-SendThread(dscn11:2181)] zookeeper.ClientCnxn: Socket connection established to dscn11/10.11.2.4:2181, initiating session
2018-09-04 00:02:10,666 INFO  [main-SendThread(dscn11:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn11/10.11.2.4:2181, sessionid = 0x656ba3c1b1c04f, negotiated timeout = 40000
2018-09-04 00:02:10,714 INFO  [main] zookeeper.ZooKeeper: Session: 0x656ba3c1b1c04f closed
2018-09-04 00:02:10,714 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2018-09-04 00:02:10,714 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:02:10,715 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:02:10,717 INFO  [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:02:10,717 INFO  [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
2018-09-04 00:02:10,719 INFO  [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bda7, negotiated timeout = 40000
2018-09-04 00:02:10,752 INFO  [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bda7 closed
Summary:
Table DBN_ANE is okay.Number of regions: 147Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_CAE is okay.Number of regions: 100Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_DHL is okay.Number of regions: 100Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn35,16020,1534221202496
Table DBN_EMS is okay.Number of regions: 226Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,15342212024962018-09-04 00:02:10,754 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut downTable DBN_GTO is okay.Number of regions: 51Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 dscn35,16020,1534221202496
Table DBN_JBD is okay.Number of regions: 238Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_KYE is okay.Number of regions: 51Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_STO is okay.Number of regions: 440Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_SUR is okay.Number of regions: 100Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_TNT is okay.Number of regions: 100Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_UPS is okay.Number of regions: 100Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_YTO is okay.Number of regions: 638Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_ZJS is okay.Number of regions: 102Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_ZTO is okay.Number of regions: 867Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_ANE is okay.Number of regions: 43Deployed on:  dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_CAE is okay.Number of regions: 21Deployed on:  dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_DHL is okay.Number of regions: 21Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_EMS is okay.Number of regions: 90Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_GTO is okay.Number of regions: 21Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn35,16020,1534221202496
Table SJYB_JBD is okay.Number of regions: 46Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn27,16020,1535964784402 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn35,16020,1534221202496
Table SJYB_KYE is okay.Number of regions: 21Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn31,16020,1529215376767 dscn35,16020,1534221202496
Table SJYB_STO is okay.Number of regions: 111Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_SUR is okay.Number of regions: 21Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn35,16020,1534221202496
Table SJYB_TNT is okay.Number of regions: 21Deployed on:  dscn11,16020,1535969281236 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn26,16020,1535018986330 dscn31,16020,1529215376767 dscn34,16020,1535675408409
Table SJYB_UPS is okay.Number of regions: 21Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn29,16020,1535540080040 dscn31,16020,1529215376767
Table SJYB_YTO is okay.Number of regions: 111Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_ZJS is okay.Number of regions: 21Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_ZTO is okay.Number of regions: 137Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_TNTIndex is okay.Number of regions: 1Deployed on:  dscn34,16020,1535675408409
Table SJYB_KJ is okay.Number of regions: 21Deployed on:  dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn29,16020,1535540080040 dscn34,16020,1535675408409
Table SJYB_PJ is okay.Number of regions: 33Deployed on:  dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn35,16020,1534221202496
Table SJYB_SF is okay.Number of regions: 92Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_UC is okay.Number of regions: 36Deployed on:  dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109
Table DBN_DEPPONIndex is okay.Number of regions: 1Deployed on:  dscn20,16020,1529215363444
Table DBN_SURIndex is okay.Number of regions: 1Deployed on:  dscn14,16020,1535088098764
Table DBN_BSHTIndex is okay.Number of regions: 2Deployed on:  dscn31,16020,1529215376767
Table DBN_APEX is okay.Number of regions: 200Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_BSHT is okay.Number of regions: 439Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_TTKD is okay.Number of regions: 267Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_ZGYZ is okay.Number of regions: 302Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table hbase:meta is okay.Number of regions: 1Deployed on:  dscn35,16020,1534221202496
Table DBN_ANEIndex is okay.Number of regions: 1Deployed on:  dscn13,16020,1529215824290
Table DBN_DEPPON is okay.Number of regions: 145Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_DHLIndex is okay.Number of regions: 1Deployed on:  dscn26,16020,1535018986330
Table DBN_SFIndex is okay.Number of regions: 2Deployed on:  dscn17,16020,1534932216753
Table DBN_YTOIndex is okay.Number of regions: 2Deployed on:  dscn30,16020,1533181074642
Table DBN_FEDEX is okay.Number of regions: 100Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_YUNDA is okay.Number of regions: 499Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_APEX is okay.Number of regions: 21Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn32,16020,1535798049804
Table SJYB_BSHT is okay.Number of regions: 118Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_TTKD is okay.Number of regions: 86Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_ZGYZ is okay.Number of regions: 38Deployed on:  dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409
Table DBN_CAEIndex is okay.Number of regions: 1Deployed on:  dscn20,16020,1529215363444
Table BOX is okay.Number of regions: 141Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_JBDIndex is okay.Number of regions: 2Deployed on:  dscn14,16020,1535088098764
Table DBN_TTKDIndex is okay.Number of regions: 1Deployed on:  dscn20,16020,1529215363444
Table hbase:namespace is okay.Number of regions: 1Deployed on:  dscn12,16020,1533367433799
Table DBN_KJIndex is okay.Number of regions: 1Deployed on:  dscn21,16020,1529215364453
Table DBN_ZJSIndex is okay.Number of regions: 1Deployed on:  dscn18,16020,1535022020180
Table DBN_UCIndex is okay.Number of regions: 1Deployed on:  dscn14,16020,1535088098764
Table DBN_PJIndex is okay.Number of regions: 1Deployed on:  dscn18,16020,1535022020180
Table DBN_FEDEXIndex is okay.Number of regions: 2Deployed on:  dscn26,16020,1535018986330
Table DBN_ZGYZIndex is okay.Number of regions: 1Deployed on:  dscn32,16020,1535798049804
Table DBN_ZTOIndex is okay.Number of regions: 2Deployed on:  dscn23,16020,1529215366710 dscn33,16020,1535968743109
Table SJYB_DEPPON is okay.Number of regions: 33Deployed on:  dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn35,16020,1534221202496
Table DBN_APEXIndex is okay.Number of regions: 1Deployed on:  dscn16,16020,1534933465007
Table DBN_EMSIndex is okay.Number of regions: 2Deployed on:  dscn15,16020,1529371666615
Table DBN_YUNDAIndex is okay.Number of regions: 2Deployed on:  dscn13,16020,1529215824290
Table SJYB_FEDEX is okay.Number of regions: 21Deployed on:  dscn11,16020,1535969281236 dscn13,16020,1529215824290 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409
Table DBN_STOIndex is okay.Number of regions: 2Deployed on:  dscn24,16020,1535104303466
Table SJYB_YUNDA is okay.Number of regions: 102Deployed on:  dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_KJ is okay.Number of regions: 119Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_PJ is okay.Number of regions: 164Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_SF is okay.Number of regions: 457Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_UC is okay.Number of regions: 210Deployed on:  dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_UPSIndex is okay.Number of regions: 2Deployed on:  dscn15,16020,1529371666615
5 inconsistencies detected.
Status: INCONSISTENT
2018-09-04 00:02:10,770 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
2018-09-04 00:02:10,771 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2656ba391b6bd9a
2018-09-04 00:02:10,772 INFO  [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bd9a closed
2018-09-04 00:02:10,773 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
[root@dscn12 ~]#

==原因分析==

从hbase hbck检查结果中可以看到以下关键信息

ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta

ERROR: There is a hole in the region chain between 97 and 97581111303371.  You need to create a new .regioninfo and region dir in hdfs to plug the hole.

错误提示了hbase meta数据存在问题，缺失的rowkey范围边界，也就是hole（Region之间有空隙，正常的Region之间是无交叉、无缝隙的）。

导致HBase整个集群状态出现问题，影响Storm的读写处理。

==解决过程==

以前有位大神给了在线hbck修复的运维命令

前提：HDFS fsck确保hbase根目录下文件没有损坏丢失，如果有，则先进行corrupt block移除。
切记：一定要在所有Region都上线之后再修复，否则修复之后可能出现重复Region。步骤1. hbase hbck 检查输出所以ERROR信息，每个ERROR都会说明错误信息。
步骤2. hbase hbck -fixTableOrphans 先修复tableinfo缺失问题，根据内存cache或者hdfs table 目录结构，重新生成tableinfo文件。
步骤3. hbase hbck -fixHdfsOrphans 修复regioninfo缺失问题，根据region目录下的hfile重新生成regioninfo文件。
步骤4. hbase hbck -fixHdfsOverlaps 修复region重叠问题，merge重叠的region为一个region目录，并从新生成一个regioninfo。
步骤5. hbase hbck -fixHdfsHoles 修复region缺失，利用缺失的rowkey范围边界，生成新的region目录以及regioninfo填补这个空洞。
步骤6. hbase hbck -fixMeta 修复meta表信息，利用regioninfo信息，重新生成对应meta row填写到meta表中，并为其填写默认的分配regionserver。
步骤7. hbase hbck -fixAssignments 把这些offline的region触发上线，当region开始重新open 上线的时候，会被重新分配到真实的RegionServer上 , 并更新meta表上对应的行信息。

我尝试执行了两个命令：

hbase hbck -fixHdfsHoles
hbase hbck -fixAssignments -fixMete

重新运行hbase hbck之后，集群恢复了正常。

==经验总结==

hbase hbck检查发现集群不健康的时候，

尝试确诊不健康的原因（有hole？regioninfo缺失？或是tableinfo缺失？）

选择适合的在线修复命令修复HBase，

修复之后，再次执行hbase hbck命令，

当检查结果为下面这种状态时，hbase为健康状态：

（HBase集群不需要重启，因为是在线修复命令）

--END--

转载于:https://www.cnblogs.com/quchunhui/p/9583746.html

一次HBase问题的解决过程（Status: INCONSISTENT）相关推荐

  1. 生产中NFS案例记录---写入权限解决过程

        生产中NFS案例记录---写入权限解决过程 NFS配置要求: 1. 将oracle文件写入到NFS Server端,注意权限要与oracle端一致. 2. Oracle端目录文件所属用户为or ...

  2. vb.net服务器启动后cpu占用了70_记一次服务器被异常程序占用的解决过程（怀疑黑客攻击）...

    最近在跑实验,但是突然发现程序运行变慢,然后top命令查看程序运行情况,发现有异常进程,名字叫 bash,占用 2400% CPU计算资源. 刚开始怀疑是挖矿程序,因实验室网络IP为教育网公网,怀疑被 ...

  3. Hbase报错解决方法ERROR: KeeperErrorCode = NoNode for /hbase/master

    步骤 今天搭建Hbase环境,遇到了此bug.一开始搭建hdfs的时候还是很顺利的,参考着别人的博客,一步一的做下去,最后可以正常启动.而在搭建HBase的时候遇到了一个很大的bug.但是造成该bug ...

  4. 记一次rc.local中python脚本无法运行的解决过程

    记一次rc.local中python脚本无法运行的解决过程 问题记录: 解决过程: 1. 检查/etc/rc.local的权限 2. 看运行出错日志 3. 修改文件不重启啊(用户切换到root了,我再 ...

  5. Oracle EBS R12 On Oracle Enterprice 5.2 安装失败 分析和解决过程

    分析和解决方法看最后的说明 [@more@] 错误如下:原因正在查找中..估计原因是创建控制文件失败.. 由于是晚上自己的安装了,所以..没注意这些 Login Page ----------- ch ...

  6. ORA-01189的完整解决过程(File is from a different RESETLOGS than previous files)

    昨天用户报告数据库不能启动了,询问用户,用户也不清楚原因.在解决过程中遇到了ORA-01189的问题,查遍了网上,包括metalink,也没有找到合适的解决方案,差点就放弃了......还好,根据OR ...

  7. 生产环境GC故障解决过程记录

    2019独角兽企业重金招聘Python工程师标准>>> 排查了五六个小时,终于解决了GC的问题,记录一下希望可以帮到有需要的人,本文假定读者对GC知识有一定的了解 如果不了解可以先参 ...

  8. Cisco 3560 丢失 IOS 解决过程

    Cisco 3560 丢失 IOS 解决过程 Flash 被清空的情况下,只能使用 Xmodem 导入,速度大约4k/s. 思科交换机3560 IOS 被删除,插上 console 之后,只能显示&q ...

  9. 一起因网线问题引起的网络故障的解决过程

    环境:上层交换机是思科的2960,下层交换机是一台普通的tp的百兆交换机,两台交换机在两个办公室,中间隔了一条道,网线走的比较麻烦,需要穿道还需要上屋顶. 问题:早上查看交换机时发现2960上对应的端 ...

  10. ssh 远程登陆异常SSH_EXCHANGE_IDENTIFICATION及解决过程

    ssh 远程登陆异常SSH_EXCHANGE_IDENTIFICATION及解决过程 参考文章: (1)ssh 远程登陆异常SSH_EXCHANGE_IDENTIFICATION及解决过程 (2)ht ...

最新文章

  1. 用C#实现在PowerPoint文档中搜索文本
  2. pandas使用replace函数替换dataframe中的值：replace函数对dataframe中指定数据列的值进行替换、替换具体数据列的相关值
  3. mysql可视化导入csv文件_我们如何将数据从.CSV文件导入MySQL表？
  4. mysql和sql互导_Mysql和SqlServer互相转换
  5. WCF与 Web Service的区别
  6. nebula-gql语句之时间差值计算
  7. 剑指 offer 树的子结构
  8. 进击吧！ Blazor ！第二期 回顾
  9. 软件构造学习笔记-第十一周
  10. tohexstring方法_Java Float类toHexString（）方法的示例
  11. 飞龙的程序员书单 – Linux
  12. 解决无线网连不上的问题
  13. CAD盗图木马分析与处置策略
  14. Webx系列之文件上传
  15. 视频号领域发布的不同时间，视频号上热门秘诀：国仁楠哥
  16. 拿Python当作计算器
  17. android开发中论坛帖子图片的排版
  18. 揭秘 zCloud 3.0丨企业需要怎么样的DBA？
  19. 运营数据分析模型—用户分层分析
  20. 开关电源的电流多大怎么选择

热门文章

  1. 并发编程之美-终章chat
  2. win7IE10和win7IE11浏览器更新所需补丁全套【附安装流程】
  3. 使用链接时间代码生成
  4. windos读写ext3工具_Windows读写Ext3文件系统
  5. android游戏开发方向初探
  6. “猜猜红桃A在哪里”——android小游戏（入门学习必备）
  7. JAVA集合容器源码解析
  8. Cadence orcad 使用MySQL搭建元件数据库及实例数据库下载
  9. jed后缀是什么文件？什么作用呢？
  10. Git学习笔记之廖雪峰Git教程