WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Region was hijacked
转自:http://mail-archives.apache.org/mod_mbox/hbase-issues/201112.mbox/%3C7071407.41918.1324627472698.JavaMail.tomcat@hel.zones.apache.org%3E
Two adjacent assignments lead region is in PENDING_OPEN state and block table disable and enable actions. ---------------------------------------------------------------------------------------------------------Key: HBASE-5092URL: https://issues.apache.org/jira/browse/HBASE-5092Project: HBaseIssue Type: BugComponents: master, regionserverAffects Versions: 0.92.0Reporter: Liu JiaAssignee: Liu JiaRegion is in PENDING_OPEN state and disable and enable are blocked.We occasionally find if two assignments which have a short interval time will lead to a PENDING_OPEN state staying in the regionInTransition map and blocking the disable and enable table actions.We found that the second assignment will set the zknode of this region to M_ZK_REGION_OFFLINE then set the state in assignmentMananger's regionInTransition map to PENDING_OPEN and abort its further operation because of finding the the region is already in the regionserver by a RegionAlreadyInTransitionException. At the same time the first assignment is tickleOpening and find the version of the zknode is messed up by the second assignment, so the OpenRegionHandler print out the following two lines:{noformat} 2011-12-23 22:12:15,197 WARN [RS_OPEN_REGION-data16,59892,1324649528415-0] zookeeper.ZKAssign(788): regionserver:59892-0x1346b43b91e0002 Attempt to transition the unassigned node for 15237599c632752b8cfd3d5a86349768 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING failed, the node existed but was version 2 not the expected version 1 2011-12-23 22:12:15,197 WARN [RS_OPEN_REGION-data16,59892,1324649528415-0] handler.OpenRegionHandler(403): Failed refreshing OPENING; region=15237599c632752b8cfd3d5a86349768, context=post_region_open {noformat} After that it tries to turn the state to FAILED_OPEN, but also failed due to wrong version,this is the output:{noformat} 2011-12-23 22:12:15,199 WARN [RS_OPEN_REGION-data16,59892,1324649528415-0] zookeeper.ZKAssign(812): regionserver:59892-0x1346b43b91e0002 Attempt to transition the unassigned node for 15237599c632752b8cfd3d5a86349768 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN failed, the node existed but was in the state M_ZK_REGION_OFFLINE set by the server data16,59892,1324649528415 2011-12-23 22:12:15,199 WARN [RS_OPEN_REGION-data16,59892,1324649528415-0] handler.OpenRegionHandler(307): Unable to mark region {NAME => 'table1,,1324649533045.15237599c632752b8cfd3d5a86349768.', STARTKEY => '', ENDKEY => '', ENCODED => 15237599c632752b8cfd3d5a86349768,} as FAILED_OPEN. It's likely that the master already timed out this open attempt, and thus another RS already has the region. {noformat} So after all that, the PENDING_OPEN state is left in the assignmentMananger's regionInTransition map and none will deal with it further, This kind of situation will wait until the master find the state out of time.The following is the test code:{code:title=test.java|borderStyle=solid} @Testpublic void testDisableTables() throws IOException {for (int i = 0; i < 20; i++) {HTableDescriptor des = admin.getTableDescriptor(Bytes.toBytes(table1));List<HRegionInfo> hris = TEST_UTIL.getHBaseCluster().getMaster().getAssignmentManager().getRegionsOfTable(Bytes.toBytes(table1));TEST_UTIL.getHBaseCluster().getMaster().assign(hris.get(0).getRegionName());TEST_UTIL.getHBaseCluster().getMaster().assign(hris.get(0).getRegionName());admin.disableTable(Bytes.toBytes(table1));admin.modifyTable(Bytes.toBytes(table1), des);admin.enableTable(Bytes.toBytes(table1));}}{code}To fix this,we add a line to public static int ZKAssign.transitionNode() to make endState.RS_ZK_REGION_FAILED_OPEN transition pass.{code:title=ZKAssign.java|borderStyle=solid}if((!existingData.getEventType().equals(beginState))//add the following line to make endState.RS_ZK_REGION_FAILED_OPEN transition pass.&&(!endState.equals(endState.RS_ZK_REGION_FAILED_OPEN))) {LOG.warn(zkw.prefix("Attempt to transition the " +"unassigned node for " + encoded +" from " + beginState + " to " + endState + " failed, " +"the node existed but was in the state " + existingData.getEventType() +" set by the server " + serverName));return -1;} {code}Run the test case again we found that before the first assignment trans the state from offline to opening, the second assignment could set the state to offline again and messed up the version of zknode.In OpenRegionHandler.process() the following part failed and make the process() return. {code:title=OpenRegionHandler.java|borderStyle=solid}if (!transitionZookeeperOfflineToOpening(encodedName,versionOfOfflineNode)) {LOG.warn("Region was hijacked? It no longer exists, encodedName=" +encodedName);return; {code} }//So we add the following code to the part to make this open region process to FAILED_OPEN.{code:title=OpenRegionHandler.java|borderStyle=solid}if (!transitionZookeeperOfflineToOpening(encodedName,versionOfOfflineNode)) {LOG.warn("Region was hijacked? It no longer exists, encodedName=" +encodedName);tryTransitionToFailedOpen(regionInfo);return;} {code}After the two amendments, two adjacent assignments will not lead to an unhandled PENDING_OPEN state.
WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Region was hijacked相关推荐
- org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online问题(暂时没有解决)
在试图使用 sqoop从mysql导数据到hbase的过程中,发生下面报错: 2020-06-30 11:53:38,636 ERROR [main] tool.ImportTool (ImportT ...
- 解决Hbase输入list,创建表报错,输入status可以,org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: S
今天恢复了一下快照,然后hbase出现了问题 2019-08-25 00:59:55,810 INFO [RpcServer.default.FPBQ.Fifo.handler=28,queue=1, ...
- org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
hbase 进行 javaAPI 操作时报了如下错误 : org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after ...
- HBASE启动失败,Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
Master日志错误: 2015-12-02 06:34:32,394 ERROR [main] master.HMasterCommandLine: Master exiting java.lang ...
- 【Hbase】报错org.apache.hadoop.hbase.RegionTooBusyException
执行了一个spark-submit程序,操作hbase但是报错org.apache.hadoop.hbase.RegionTooBusyException,具体错误如下 17/12/07 11:49: ...
- hbase报错:org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
报错 hbase(main):002:0> status 1 active master, 0 backup masters, 2 servers, 0 dead, 0.0000 average ...
- org.apache.hadoop.hbase.PleaseHoldException: Master is initializing(解決方案汇总+自己摸索)
#--------------------------------------环境(start)---------------------------------------------------- ...
- 【HBase】关于包org.apache.hadoop.hbase.client
Package org.apache.hadoop.hbase.client 提供HBase客户端接口 参考: Description Interface概要 Interface Descript ...
- 【异常】org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
[异常]org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions: ...
最新文章
- java 输入框输入1到9_java程序设计 习题答案1到9
- android学习笔记九——RatingBar
- 服务器端使用jsp还是asp_快速了解 ASP.NET Core Blazor
- 新书《活用UML-需求分析高手》详细大纲(持续更新中)
- oracle客户端ora 12541,Oracle 11g 64bit下程序报ORA-12541: TNS: 无监听程序解决办法
- iOS动画系列之四:基础动画之平移篇
- SpringBoot学习---SpringSecurity与Shiro
- python实现九九乘法表代码解释_python编写九九乘法表代码
- mmd动作:Bad End Night
- python修改文件的某一行_简单文件操作python 修改文件指定行的方法
- Verilog(2):与或非运算
- 如何检测猥琐的私有SDWAN隧道协议
- VS 错误: cout 不明确
- 超市会员管理系统(面向对象)
- php漂浮,【飘】【漂】:【飘浮】【漂浮】、【漂泊】【飘泊】【飘薄】
- form-data和x-www-form-urlencoded的区别和延伸
- 假如我有一台新电脑, 我要做些什么?
- 必应API接口node.js版 - 极客玩家大白
- 在字符串两侧填充指定字符ljust()与rjust()方法
- linux menuconfig搜索,Linux make menuconfig查找并快速跳转指定驱动选项