Hadoop常见异常及其解决方案
1、Shell$ExitCodeException
现象:运行hadoop job时出现如下异常:
14/07/09 14:42:50 INFO mapreduce.Job: Task Id : attempt_1404886826875_0007_m_000000_1, Status : FAILED
Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container exited with a non-zero exit code 1
原因及解决办法:原因未知。重启可恢复正常
2、libhadoop.so.1.0.0 which might have disabled stack guard
现象:Hadoop 2.2.0 - warning: You have loaded library /home/hadoop/2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard.
export PATH=$PATH:/home/jediael/hadoop-2.4.1/bin:/home/jediael/hadoop-2.4.1/sbin
export HADOOP_HOME=/home/jediael/hadoop-2.4.1
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
3、Retrying connect to server: master166/10.252.48.166:9000. Already tried 0 time(s)
在datanode上执行hdfs相关命令时,出现以下错误:
[jediael@slave156 ~]$ hadoop fs -ls /
14/08/31 15:00:37 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/08/31 15:00:38 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/08/31 15:00:39 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/08/31 15:00:40 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/08/31 15:00:41 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/08/31 15:00:42 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/08/31 15:00:43 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/08/31 15:00:44 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/08/31 15:00:45 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/08/31 15:00:46 INFO ipc.Client: Retrying connect to server: master166/10.252.48.166:9000. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
ls: Call to master166/10.252.48.166:9000 failed on connection exception: java.net.ConnectException: Connection refused
出现以上错误,通常都是由于datanode无法连接到namenode所致,以下是一种情况:
/etc/hosts中存在127.0.0.1 *****的配置,如
127.0.0.1 localhost
将这些配置去掉,然后重新格式化namenode,并重启hadoop进程即可解决。
或者是以下原因:
hadoop安装完成后,必须要用haddop namenode format格式化后,才能使用,如果重启机器
在启动hadoop后,用hadoop fs -ls命令老是报 10/09/25 18:35:29 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 0 time(s).的错误,
用jps命令,也看不不到namenode的进程, 必须再用命令hadoop namenode format格式化后,才能再使用
原因是:hadoop默认配置是把一些tmp文件放在/tmp目录下,重启系统后,tmp目录下的东西被清除,所以报错
解决方法:在conf/core-site.xml 中增加以下内容
<property>
<name>hadoop.tmp.dir</name>
<value>/var/log/hadoop/tmp</value>
<description>A base for other temporary directories</description>
</property>
重启hadoop后,格式化namenode即可
4、Permission denied: user=liaoliuqing, access=WRITE, inode="":jediael:supergroup:rwxr-xr-x
原因为用户权限不足,能能访写HDFS中的文件。
解决方案:
关闭hadoop权限,在hdfs-site.xml文件中添加
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
5、Incompatible namespaceIDs
2015-02-02 15:10:57,526 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-02-02 15:10:57,543 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2015-02-02 15:10:57,543 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-02-02 15:10:57,544 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
2015-02-02 15:10:57,699 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2015-02-02 15:10:58,090 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /mnt/tmphadoop/dfs/data: namenode namespaceID = 2017454015; datanode namespaceID = 1238467850
at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:414)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:321)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1712)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1651)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1669)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1795)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1812)
问题原因:
每次namenode format会重新创建一个namenodeId,而${hadoop.tmp.dir}/dfs/data下包含了上次format下的id,当重新执行namenode format时清空了namenode下的数据,但是没有清空datanode下的数据,所以造成namenode节点上的namespaceID与 datanode节点上的namespaceID不一致,从而导致从现上述异常,启动失败。
解决办法:
(1)停止hadoop
stop-all.sh
(2)在各个slave中删除dfs.data.dir中的内容。若此属性未修改,则其默认值为
<property>
<name>${dfs.data.dir}</name>
<value>${hadoop.tmp.dir}/dfs/data</value>
<description>Determines where on the local filesystem an DFS data node
should store its blocks. If this is a comma-delimited
list of directories, then data will be stored in all named
directories, typically on different devices.
Directories that do not exist are ignored.
</description>
</property>
(3)重新格式化namenode
hadoop namenode -format
然后start-all.sh启动hadoop
以上解决办法需要将原有数据删除,若数据不能删除,则使用以下方法之一:
(1)修改${dfs.data.dir}/current/VERSION文件,将datanode中的id改成与namenode中的id一致。
(2)修改${dfs.data.dir}
Hadoop常见异常及其解决方案相关推荐
- Hadoop常见异常
Hadoop常见异常 @(HADOOP)[hadoop] Hadoop常见异常 1ShellExitCodeException 2libhadoopso100 which might have dis ...
- Hadoop常见错误及解决方案、Permission denied: user=dr.who, access=WRITE, inode=“/“:summer:supergroup:drwxr-xr-x
文章目录 4.常见错误及解决方案 1)防火墙没关闭.或者没有启动YARN 7)不识别主机名称 8)DataNode和NameNode进程同时只能工作一个. 9)执行命令不生效,粘贴Word中命令时,遇 ...
- ASP.NET MVC开发中常见异常及解决方案
NHibernate:no persister for 异常 1.配置文件后缀名写错 mapping file 必须是.hbm.xml结尾 2.Web.config配置里面引用实体 <sessi ...
- 绝地求生服务器维护错误,哪些是绝地求生常见异常解决方案?
现在一款黑马游戏当属绝地求生,对于绝地求生常见异常问题,腾讯电脑管家官网也整理出一套方案,具体是什么方案?能够解决绝地求生常见异常吗?接下来和小编一起来看看吧. 主要症状 绝地求生玩不了,出现各种错误 ...
- redis 缓存预热_Redis异常问题解决方案:缓存雪崩、预热、穿透、击穿、降解方案分析...
不管你在工作中还是面试中,Redis异常问题的解决方案都是不得不去学习了解的,下面来细品一下! 一.缓存雪崩 1.什么是缓存雪崩? 如果缓存集中在一段时间内失效,发生大量的缓存穿透,所有的查询都落在数 ...
- Hadoop中的MapReduce框架原理、数据清洗(ETL)、MapReduce开发总结、常见错误及解决方案
文章目录 13.MapReduce框架原理 13.7 数据清洗(ETL) 13.7.1 需求 13.7.1.1 输入数据 13.7.1.2 期望输出数据 13.7.2 需求分析 13.7.3实现代码 ...
- 大数据技术之Hadoop(Hadoop企业优化常见错误及解决方案)
6.1 MapReduce 跑的慢的原因 6.2 MapReduce优化方法 MapReduce优化方法主要从六个方面考虑:数据输入.Map阶段.Reduce阶段.IO传输.数据倾斜问题和常用的调优参 ...
- Selenium常见异常分析及解决方案
Selenium常见异常分析及解决方案 参考文章: (1)Selenium常见异常分析及解决方案 (2)https://www.cnblogs.com/superhin/p/11454861.html ...
- Linux MySQL 常见无法启动或启动异常的解决方案
Linux MySQL 常见无法启动或启动异常的解决方案 参考文章: (1)Linux MySQL 常见无法启动或启动异常的解决方案 (2)https://www.cnblogs.com/youjia ...
最新文章
- 用TCP/IP进行网际互联一
- 合川专升本计算机怎么学升学率,2019年重庆市合川太和中学高考喜报
- CentOS7之新建用户与SSH登陆
- 第一个小项目 - 去掉指定文件中的注释
- POJ2104 K-th Number [分块做法]
- 学习 WCF (5)--学习Binding的使用
- 3.PHP核心技术与最佳实践 --- PHP网络技术及应用(cURL,socket,SMTP,WebService,Cookie,Session,localStorage)
- 波普尔心智格列高利心智_心智与人工智能理论
- 最全解读Unicode字符编码
- gps模拟器 matlab,基于Matlab/Simulink的数字中频GPS信号软件模拟器
- 安全认证框架之Shiro详解
- AI眼中的世界 ——人工智能绘画进阶
- 一名全栈工程师的必备工具箱
- 艺术家艾·克里斯汀·麦克拉维·佩恩gp采访23
- JavaScript中阻止事件冒泡的浏览器兼容
- 回顾2017展望未来
- 比较好的运动耳机,好用的运动耳机推荐
- IOU破局之路 | Focal EIOU:打破IOU/GIOU/CIOU的局限
- 计算机操作系统学习笔记----进程管理
- uniapp 使用map组件 动态自定义标记点图标及内容文字
热门文章
- 【最贴心】C++字符串转换(stoi;stol;stoul;stoll;stoull;stof;stod;stold)
- kali如何取得超级用户权限_如何在 Ubuntu 上为用户授予和移除 sudo 权限 | Linux 中国...
- 天津理工大学c语言上机报告3,天津理工大学-c语言上机报告4.pdf
- shell函数与数组(了解函数,递归函数运用,了解数组,创建基数和偶数数组,冒泡排序,调试命令)
- 在表示计算机内存储器容量时 1gb等于,在表示计算机内存储器容量时,1GB等于_________MB...
- Ubuntu安装中文语言包
- 关于堆空间溢出的错误解决办法
- php rdkafka实例,kafka 消息队列 php-rdkafka扩展示例
- php群发不用foreach,如何在没有foreach的情况下使用PHP生成器?
- win7系统服务器环境配置,windows7系统安装与配置Tomcat服务器环境