HIVE2:ERROR [main]: ql.Driver (:()) - FAILED: Execution Error, return code 1 from org.apache.hadoop.

在Hive2.1 on Tez环境中运行select count(*) from students;时，遇到

ERROR [main]: ql.Driver (:()) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask

查看hive日志，具体问题是：

2016-12-20T17:38:50,298 INFO [Tez session start thread]: client.TezClient (:()) - Could not connect to AM, killing session via YARN, sessionName=HIVE-c6220a27-a8a5-4d55-a36d-a6d87bd8931c, applicationId=application_1482283108292_0001
2016-12-20T17:38:50,313 INFO [Tez session start thread]: impl.YarnClientImpl (:()) - Killed application application_1482283108292_0001
2016-12-20T17:38:50,313 ERROR [Tez session start thread]: tez.TezSessionState (:()) - Failed to start Tez session
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1482283108292_0001 failed 2 times due to AM Container for appattempt_1482283108292_0001_000002 exited with exitCode: -103

For more detailed output, check application tracking page:http://master:8088/cluster/app/application_1482283108292_0001Then, click on links to logs of each attempt.
Diagnostics: Container [pid=3048,containerID=container_1482283108292_0001_02_000001] is running beyond virtual memory limits. Current usage: 139.1 MB of 1 GB physical memory used;2.6 GB of 2.1 GB virtual memory used. Killing container.

解决方法：

调整hadoop配置文件yarn-site.xml中值：

<property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>2048</value>
    <description>default value is 1024</description>
</property>
<property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>2.1</value>
    <description>default value is 2.1</description>
</property>

增加yarn.scheduler.minimum-allocation-mb 数量，从缺省1024改为2048；上述运行问题即刻得到解决；

单独调整yarn.nodemanager.vmem-pmem-ratio从缺省值2.1调整到3.0，从计算上Vm=3.0*1=3.0>2.6 ，但是上述问题依然存在（原因？）

原因：

该错误是YARN的虚拟内存计算方式导致，上例中用户程序申请的内存为1Gb，YARN根据此值乘以一个比例（默认为2.1）得出申请的虚拟内存的值，当YARN计算的用户程序所需虚拟内存值大于计算出来的值时，就会报出以上错误。调节比例值可以解决该问题。具体参数为：yarn-site.xml中的yarn.nodemanager.vmem-pmem-ratio

HIVE2:ERROR [main]: ql.Driver (:()) - FAILED: Execution Error, return code 1 from org.apache.hadoop.相关推荐

执行HiveSql时报错“FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTa”
执行HiveSql时报错"FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapR ...
Hive任务执行报错：FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
报错内容如下: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask 22/ ...
报错：FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask原因查找
项目场景: 使用hive分析.处理数据时问题描述执行hive语句后,报错:FAILED: Execution Error, return code 2 from org.apache.hadoop ...
hive遇到FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask错误...
hive遇到FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask错误起因 ...
Hive创表异常，FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Me taException(m ...
hive 运行命令后FAILED: Execution Error, return code 137 from org.apache.hadoop.hive.ql.ex ec.mr.MapredLoc
FAILED: Execution Error, return code 137 from org.apache.hadoop.hive.ql.ex ec.mr.MapredLocalTask vi ...
Hive更改表名FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Una ble
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Una ble to alter ...
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. ORC split
文章目录 1. 报错 2. 解决方法 1. 报错 FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec. ...
Hive报错FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
Hive报错Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoo ...

HIVE2:ERROR [main]: ql.Driver (:()) - FAILED: Execution Error, return code 1 from org.apache.hadoop.

HIVE2:ERROR [main]: ql.Driver (:()) - FAILED: Execution Error, return code 1 from org.apache.hadoop.相关推荐

最新文章

热门文章