hive异常 return code X from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask 解决
转自:https://www.iteye.com/blog/vase-2090320
不知道是不是hive-0.12版增强了local mode的原因,在之前版本运行好好的Hive-QL在这个版本上错误频频,折磨一天多以后终于定位到原因,把在内部的总结在这再记录下,希望对遇到同样问题的筒子们有所帮助。
部分一 关于return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
Hive升级到0.12版之后,若干原来在0.10上执行正常的SQL会在新版上报错误 “return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask”,查看hive执行日志,从中找到如下错误
Total MapReduce jobs = 1
java.io.IOException: Cannot run program "/data/opt/hadoop_cdh5/bin/hadoop" (in directory "/root"): error=13, 权限不够 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041) at java.lang.Runtime.exec(Runtime.java:617) at java.lang.Runtime.exec(Runtime.java:450) at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.execute(MapredLocalTask.java:253) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1485) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1263) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1091) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:921) at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:198) at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:644) at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:628) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: error=13, 权限不够 at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:135) at java.lang.ProcessImpl.start(ProcessImpl.java:130) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022) ... 19 more
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
从上边错误及下边报错的类MapredLocalTask可以看出跟本地任务有关
hive从0.7版以后,为了提高小数据的计算速度,增加了本地模式,即将hdfs上的数据拉到hiveserver本地进行计算,可以通过以下几个参数对相关行为进行设置
hive.exec.mode.local.auto=false
hive.exec.mode.local.auto.input.files.max=4
hive.exec.mode.local.auto.inputbytes.max=134217728
其中第一个为不启用本地模式,第二个参数表示文件数小于4时使用本地模式,第三个参数表示文件大小小于128m时采用本地模式
默认为不启用本地模式;在启用的情况下,满足第二、三个条件中的任意一个都会使用本地模式。
在之前我们用过的0.8.1、0.10版上都未遇到过上述错误,怀疑是现在0.12版本的问题突然导致上述错误。任务是在root用户下通过crontab调用的,进入shell后先启动hiveserver,所以默认工作目录其实是/root;为了能正常读写hdfs上的文件,hiveserver在启动时切换到了hdfs用户,一旦遇到上述两种满足启用本地模式的情况,hdfs用户试图向当前工作目录/root拉取数据,必然没有权限从而导致以上错误。
理清问题所在就好办了,我们可以先创建一个目录,把用户、用户组授权给hdfs,进入shell后,先切换工作目录,然后再启动hiveserver即可。如hdfs的home目录/home/hdfs
然后在任务shell的公共配置文件conf/kettle.conf中增加一行切换目录脚本即可解决以上问题
cd /home/hdfs
部分二 关于return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask 类似上一篇中return code 1的问题,这个也是跟hive本地任务有关系。
从hive的日志中可以找到出错时本地日志文件,如下:
查看日志文件内容
2014-07-10 11:50:37,606 INFO mr.ExecDriver (SessionState.java:printInfo(417)) - Execution log at: /tmp/hdfs/hdfs_20140710114949_ab4d1d02-0637-4abd-9e45-2a27c5d740d9.log
2014-07-10 11:50:37,711 WARN conf.Configuration (Configuration.java:loadProperty(2358)) - file:/tmp/hdfs/hive_2014-07-10_11-49-37_877_2428431256361163465-1/-local-10009/jobconf
2014-07-10 11:50:37,720 WARN conf.Configuration (Configuration.java:loadProperty(2358)) - file:/tmp/hdfs/hive_2014-07-10_11-49-37_877_2428431256361163465-1/-local-10009/jobconf
2014-07-10 11:50:37,798 INFO log.PerfLogger (PerfLogger.java:PerfLogBegin(97)) - <PERFLOG method=deserializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
2014-07-10 11:50:37,798 INFO exec.Utilities (Utilities.java:deserializePlan(732)) - Deserializing MapredLocalWork via kryo
2014-07-10 11:50:38,043 INFO log.PerfLogger (PerfLogger.java:PerfLogEnd(124)) - </PERFLOG method=deserializePlan start=1404964237798 end=1404964238043 duration=245 from=org.apa
2014-07-10 11:50:38,050 INFO mr.MapredLocalTask (SessionState.java:printInfo(417)) - 2014-07-10 11:50:38 Starting to launch local task to process map join; maximum memory =
2014-07-10 11:50:38,059 INFO mr.MapredLocalTask (MapredLocalTask.java:initializeOperators(389)) - fetchoperator for t2:t_tmp_user_first_login created
2014-07-10 11:50:38,198 INFO exec.TableScanOperator (Operator.java:initialize(338)) - Initializing Self 0 TS
2014-07-10 11:50:38,198 INFO exec.TableScanOperator (Operator.java:initializeChildren(403)) - Operator 0 TS initialized
2014-07-10 11:50:38,199 INFO exec.TableScanOperator (Operator.java:initializeChildren(407)) - Initializing children of 0 TS
2014-07-10 11:50:38,199 INFO exec.SelectOperator (Operator.java:initialize(442)) - Initializing child 1 SEL
2014-07-10 11:50:38,199 INFO exec.SelectOperator (Operator.java:initialize(338)) - Initializing Self 1 SEL
2014-07-10 11:50:38,605 ERROR mr.MapredLocalTask (MapredLocalTask.java:executeFromChildJVM(324)) - Hive Runtime Error: Map local work failed
java.lang.RuntimeException: java.lang.ClassNotFoundException: com.renren.hive.date.GetWeekISO at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:132) at org.apache.hadoop.hive.ql.exec.FunctionRegistry.isStateful(FunctionRegistry.java:1474) at org.apache.hadoop.hive.ql.exec.FunctionRegistry.isDeterministic(FunctionRegistry.java:1437) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:132) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:83) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEval(ExprNodeEvaluatorFactory.java:73) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:377) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:453) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:409) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:188) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:377) at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:408) at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:302) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:728) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.ClassNotFoundException: com.renren.hive.date.GetWeekISO at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:270) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:130)
由上可知,这次是找不到UDF的类(如遇到其他情况,需要具体问题具体分析),虽然在进入hive的时候通过add jar语句将自定义函数的jar包添加到hadoop集群,但在本地模式时确找不到了。定位到问题就好解决了:既然是local模式找不到udf jar包,说明在add jar步骤只是向当前job在hdfs上的工作目录下添加了,无视本地工作目录;那么我们就直接把udf的jar包copy到hive的lib目录下,测 试正常。该问题在之前用过的hive 0.10、0.8.1中都未遇到过,初步猜测跟0.12版的bug有关,具体原因就需要花时间翻代码对照前后版本的变动了
从org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask代码中看,还有return code 3的情况,现在幸运的尚未遇到,遇到后再补记录
hive异常 return code X from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask 解决相关推荐
- hive3.x异常- return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
提交joinsql核心异常如下 return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask The value of pr ...
- 使用hive报 return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask解决方法
1.情况 两表join 其他时间数据正常插入.唯独插入7月1日数据时 , 报错: join 语句 insert overwrite table A partition (log_date= '2021 ...
- hive问题-return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
执行hive sql时遇到问题: FAILED:Execution Error,return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedT ...
- Hive创表异常,FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Me taException(m ...
- 成功解决: return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
异常: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop. ...
- hive -- return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
异常: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop. ...
- Hive任务执行报错:FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
报错内容如下: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask 22/ ...
- hive遇到FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask错误...
hive遇到FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask错误 起因 ...
- hive中删除表的错误Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException
1:请看操作 [jifeng@jifeng02 hive-0.12.0-bin]$ hiveLogging initialized using configuration in jar:file:/h ...
最新文章
- 让餐厅放心的云服务-雅座CRM技术解密
- Django Rest Framework 视图和路由
- 如何得到所有可视化窗口的句柄?
- metinfo mysql off_利用Sqlmap测试MetInfo企业网站管理系统MySql注入漏洞
- Solr系列二:solr-部署详解(solr两种部署模式介绍、独立服务器模式详解、SolrCloud分布式集群模式详解)...
- redmine-1.2.2安装代码评审插件
- Leetcode 538.二叉树转换为累加树
- K8S 的报错问题解决
- ArduinoUNO-IRremote 红外线接收模块使用(还没写完)
- 关于写论文的小技巧[一]:公式编号
- vue移动端项目vant组件库之style内置样式
- 第四章 姜诸儿意气风发登君位 鲁桓公窝囊枉死彭生手
- mysql 数据库第二次安装不了_mysql数据库二次安装无法启动
- [机缘参悟-66]:怎样才能让别人愿意帮你:利益共享法则、“大道”、“人性”
- 5.14——教你把ssh抄成ssm
- nofollow标签的作用 nofollow标签添加方法
- unicloud开发微商管理小程序-商品私域推广
- Mac系统下降级安装stlink-1.4.0方法
- MySQL备份之--冷备(实用工具)
- 自从会了Python之后,我就没用过PS了!3秒带你将照片变成素描图片!