file:/usr/local/hive/iotmp/89de7dfe-8f26-4a6e-87b4-bacd16c2b2c8/hive_2021-11-05_05-06-07_555

1、发现问题

场景：当多个sql脚本任务同时在运行的时候经常报如下错误。

具体报错：FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. File file:/usr/local/hive/iotmp/89de7dfe-8f26-4a6e-87b4-bacd16c2b2c8/hive_2021-11-05_05-06-07_555_3392062193706637485-1/-local-10004/HashTable-Stage-2 does not exist

INFO - MapredLocal task succeeded INFO - Launching Job 1 out of 1 INFO - Number of reduce tasks is set to 0 since there's no reduce operator INFO - Cannot run job locally: Input Size (= 6505604400) is larger than hive.exec.mode.local.auto.inputbytes.max (= 134217728) INFO - java.io.FileNotFoundException: File file:/usr/local/hive/iotmp/89de7dfe-8f26-4a6e-87b4-bacd16c2b2c8/hive_2021-11-05_05-06-07_555_3392062193706637485-1/-local-10004/HashTable-Stage-2 does not exist INFO - at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:429) INFO - at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1515) INFO - at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1555) INFO - at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:574) INFO - at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:319) INFO - at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151) INFO - at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) INFO - at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) INFO - at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183) INFO - at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839) INFO - at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526) INFO - at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237) INFO - at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227) INFO - at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) INFO - at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) INFO - at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) INFO - at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336) INFO - at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474) INFO - at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490) INFO - at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793) INFO - at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) INFO - at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) INFO - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) INFO - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) INFO - at java.lang.reflect.Method.invoke(Method.java:498) INFO - at org.apache.hadoop.util.RunJar.run(RunJar.java:221) INFO - at org.apache.hadoop.util.RunJar.main(RunJar.java:136) INFO - Job Submission failed with exception 'java.io.FileNotFoundException(File file:/usr/local/hive/iotmp/89de7dfe-8f26-4a6e-87b4-bacd16c2b2c8/hive_2021-11-05_05-06-07_555_3392062193706637485-1/-local-10004/HashTable-Stage-2 does not exist)' INFO - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. File file:/usr/local/hive/iotmp/89de7dfe-8f26-4a6e-87b4-bacd16c2b2c8/hive_2021-11-05_05-06-07_555_3392062193706637485-1/-local-10004/HashTable-Stage-2 does not exist

2、分析问题

看报错可能是hive-site配置的有问题，查看hive配置文件hive-site.xml发现下面四个参数都含有/usr/local/hive/iotmp目录，怀疑是不是因为相同目录相互影响导致的。那么看下hive官网对这四个参数的定义描述。

hive.exec.local.scratchdir

hive.downloaded.resources.dir

hive.querylog.location

hive.server2.logging.operation.log.location

看下配置文件这四个参数的配置：

 <property><name>hive.exec.local.scratchdir</name><value>/usr/local/hive/iotmp</value><description>Local scratch space for Hive jobs</description></property><property><name>hive.downloaded.resources.dir</name><value>/usr/local/hive/iotmp</value><description>Temporary local directory for added resources in the remote file system.</description></property><property><name>hive.querylog.location</name><value>/usr/local/hive/iotmp</value><description>Location of Hive run time structured log file</description></property><property><name>hive.server2.logging.operation.log.location</name><value>${/usr/local/hive/iotmp}/${system:user.name}/operation_logs</value><description>Top level directory where operation logs are stored if logging functionality is enabled</description></property>

官网参数解释

官网地址：Configuration Properties - Apache Hive - Apache Software Foundation

hive.exec.scratchdir

hive.exec.scratchdir
Default Value: /tmp/${user.name} in Hive 0.2.0 through 0.8.0; /tmp/hive-${user.name} in Hive 0.8.1 through 0.14.0; or /tmp/hive in Hive 0.14.0 and later
Added In: Hive 0.2.0; default changed in 0.8.1 and in 0.14.0 with HIVE-6847  and HIVE-8143
Scratch space for Hive jobs. This directory is used by Hive to store the plans for different map/reduce stages for the query as well as to stored the intermediate outputs of these stages.

hive.exec.local.scratchdir

hive.exec.local.scratchdir
Default Value: /tmp/${user.name}
Added In: Hive 0.10.0 with HIVE-1577
Scratch space for Hive jobs when Hive runs in local mode.  Also see hive.exec.scratchdir .

hive.querylog.location

hive.querylog.location
Hive 实时查询日志所在的目录，如果该值为空，将不创建实时的查询日志。
默认值：'/tmp/${user.name}'

hive.downloaded.resources.dir

 <property><name>hive.downloaded.resources.dir</name><!--value>${system:java.io.tmpdir}/${hive.session.id}_resources</value--><value>/user/local/hive/tmp/${hive.session.id}_resources</value><description>Temporary local directory for added resources in the remote file system.</description></property>

从官网可以看到hive.exec.scratchdir、hive.exec.local.scratchdir、hive.querylog.location这三个参数的默认值是一样，并没有看到我想要的参数默认应该不一样的结果。则继续分析推理。先按照自己的想法把参数配置成不一样的值试一下。

3、解决问题

把这几个参数改成不同的目录(是否可行有待观察)

<property><name>hive.exec.scratchdir</name><value>/tmp/hive</value><description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/&lt;username&gt; is created, with ${hive.scratch.dir.permission}.</description>
</property>
<property><name>hive.exec.local.scratchdir</name><value>/usr/local/hive/iotmp</value>
</property>
<property><name>hive.downloaded.resources.dir</name><value>/usr/local/hive/iotmp/${hive.session.id}_resources</value>
</property>
<property><name>hive.querylog.location</name><value>/usr/local/hive/iotmp/root</value>
</property>
<property><name>hive.server2.logging.operation.log.location</name><value>/usr/local/hive/iotmp/root/operation_logs</value>
</property>

改完观察了一个星期发现并没有再报错。

4、结论

在hive配置文件hive-site.xml中把这四个参数改成不同目录可以解决问题。

hive.exec.local.scratchdir

hive.downloaded.resources.dir

hive.querylog.location

hive.server2.logging.operation.log.location

file:/usr/local/hive/iotmp/89de7dfe-8f26-4a6e-87b4-bacd16c2b2c8/hive_2021-11-05_05-06-07_555_3392062相关推荐

异常处理——Logging initialized using configuration in jar:file:/usr/local/apache-hive-2.3.4/lib/hive-comm
启动hive一直: Logging initialized using configuration in jar:file:/usr/local/apache-hive-2.3.4/lib/hive- ...
/usr/bin/install: cannot create regular file ‘/usr/local/lib/libpcre.so.1.2.13‘: Permission denied
报错 /usr/bin/install: cannot create regular file '/usr/local/lib/libpcre.so.1.2.13': Permission denie ...
Starting MySQL.. ERROR! The server quit without updating PID file (/usr/local/mysql/data/vm10-0-0-19
输入:service mysqld start 报错: Starting MySQL.. ERROR! The server quit without updating PID file (/usr/ ...
mysql初始化错误【一】Can't find error-message file '/usr/local/mysql/errmsg.sys'
环境:CentOS 7.2 MySQL 5.7.18 从mysql官方网站下载rpm包到服务器本地,依次安装下面的RPM包: mysql-community-common-5.7.18-1.el7.x ...
Mysql 启动报错解析：Starting MySQL.. ERROR! The server quit without updating PID file (/usr/local/mysql/dat
现象: root@centos74 ~]# service mysqld start Starting MySQL.. ERROR! The server quit without updating ...
MySQL常见错误：Starting MySQL...The server quit without updating PID file (/usr/local/mysql/data/localhos
常见错误1:Starting MySQL-The server quit without updating PID file (/usr/local/mysql/data/localhost.loca ...
linux下安装mysql问题：mysqld_safe mysqld from pid file /usr/local/mysql/data/mysql.pid ended
原文链接 : linux下安装mysql问题总结(一)mysqld_safe mysqld from pid file /usr/local/mysql/data/mysql.pid ended li ...
启动Mysql时报错：mysqld_safe mysqld from pid file /usr/local/mysql/data/Linux.pid ended
今天启动下mysql 的安全后台启动,发现报错:mysqld_safe mysqld from pid file /usr/local/mysql/data/Linux.pid ended 原本我以为 ...
3934:C 10 Aug 2022 00:14:04.239 # Fatal error, can‘t open config file ‘/usr/local/redis/./redis-conf
报错:3934:C 10 Aug 2022 00:14:04.239 # Fatal error, can't open config file '/usr/local/redis/./redis-c ...

file:/usr/local/hive/iotmp/89de7dfe-8f26-4a6e-87b4-bacd16c2b2c8/hive_2021-11-05_05-06-07_555_3392062

file:/usr/local/hive/iotmp/89de7dfe-8f26-4a6e-87b4-bacd16c2b2c8/hive_2021-11-05_05-06-07_555_3392062相关推荐

最新文章

热门文章