CDH集群下hadoop异常PriviledgedActionException as:root (auth:SIMPLE)解决

在hadoop的应用平台看到的错误如下，根据这个错误参考了网络上的一些解决方法，没有解决这个问题，后来找了下yarn的日志

Application application_1553509480321_0011 failed 2 times due to AM Container for appattempt_1553509480321_0011_000002 exited with exitCode: -1000

For more detailed output, check application tracking page:http://cdh03:8088/proxy/application_1553509480321_0011/Then, click on links to logs of each attempt.

Diagnostics: File does not exist: hdfs://cdh03:8020/user/root/.sparkStaging/application_1553509480321_0011/__spark_conf__.zip

java.io.FileNotFoundException: File does not exist: hdfs://cdh03:8020/user/root/.sparkStaging/application_1553509480321_0011/__spark_conf__.zip

at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1269)

at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1261)

at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)

at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1261)

at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)

at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)

at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:364)

at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:362)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)

at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:361)

at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Failing this attempt. Failing the application.

yarn的日志如下：

2019-03-27 13:53:08,961 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /opt/data/yarn/nm/usercache/root/appcache/application_1553509480321_0011 = file:/opt/data/yarn/nm/usercache/root/appca
che/application_1553509480321_0011
2019-03-27 13:53:08,983 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: hdfs://cdh03:8020/user/root/.sparkStaging/applicati
on_1553509480321_0011/__spark_conf__.zip
2019-03-27 13:53:08,985 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: { hdfs://cdh03:8020/user/root/.sparkStaging/application_1553509480321_0011/__spark_conf__.zip, 1553665982289,
ARCHIVE, null } failed: File does not exist: hdfs://cdh03:8020/user/root/.sparkStaging/application_1553509480321_0011/__spark_conf__.zip
java.io.FileNotFoundException: File does not exist: hdfs://cdh03:8020/user/root/.sparkStaging/application_1553509480321_0011/__spark_conf__.zip
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1269)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1261)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1261)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:364)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:361)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

2019-03-27 13:53:08,986 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1553509480321_0011_02_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED
2019-03-27 13:53:08,986 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Container container_1553509480321_0011_02_000001 sent RELEASE event on a resource request { hdfs://cdh03:8020/u
ser/root/.sparkStaging/application_1553509480321_0011/__spark_conf__.zip, 1553665982289, ARCHIVE, null } not present in cache.
2019-03-27 13:53:08,986 WARN org.apache.hadoop.ipc.Client: interrupted waiting to send rpc request to server
java.lang.InterruptedException
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1088)
at org.apache.hadoop.ipc.Client.call(Client.java:1483)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy89.heartbeat(Unknown Source)
at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:257)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:171)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:131)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1147)
2019-03-27 13:53:08,987 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=root OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1553509480321_0011 CONTAINERID=container_1553509480321_0011_02_000001

根据“PriviledgedActionException as:root (auth:SIMPLE)”可以知道是权限的问题，参考文章：http://hadoop-common.472056.n3.nabble.com/UserGroupInformation-PriviledgedActionException-as-root-auth-SIMPLE-td4038525.html

每个NM的core-site.xml设置下下面的属性就可以了
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

CDH集群下hadoop异常PriviledgedActionException as:root (auth:SIMPLE)解决相关推荐

大规模集群下Hadoop NameNode如何承载每秒上千次的高并发访问
目录一.问题源起二.HDFS优雅的解决方案 (1)分段加锁机制 + 内存双缓冲机制 (2)多线程并发吞吐量的百倍优化 (3)缓冲数据批量刷磁盘 + 网络的优化四.总结五.参考文章一.问题源起 ...
CDH集群中HDFS单点故障解决方案：HA模式（High Availability）
CDH集群开发HDFS的High Avaliability模式 Step1 什么是单点故障? Step2 HDFS为什么会出现单点故障?它是怎么解决的? Step3 开启HDFS的高可用(High A ...
0110-如何给Kerberos环境下的CDH集群添加Gateway节点
2019独角兽企业重金招聘Python工程师标准>>> Fayson的github: https://github.com/fayson/cdhproject 1.文档编写目的 Ga ...
从零搭建生产Hadoop集群（五）——CDH集群修改主机名与IP
从零搭建生产Hadoop集群(五)--CDH集群修改主机名与IP 一.概述二.完整步骤 1.官网推荐步骤: (1)验证下SSL/TLS认证是否所有服务都有配备,确认创建新的SSL/TLS认证. (2 ...
Hadoop高可用集群下namenode格式化失败问题解决
Hadoop高可用集群下namenode格式化失败问题解决输入hdfs namenode -format报如下错误解决方法: 1.在zookeeper目录下执行./bin/zkServer.sh ...
hadoop namenode启动不了_集群版hadoop安装，写给大忙人看的
导语如果之前的单机版hadoop环境安装满足不了你,集群版hadoop一定合你胃口,轻松入手. 目录集群规划前置条件配置免密登录 3.1 生成密匙 3.2 免密登录 3.3 验证免密登录集群 ...
kill 进程_05516.1普通用户配置kill CDH集群进程权限
作者:李继武 1 文档编写目的在前面的文章<0548-5.13.1-如何使用普通用户管理CM server和agent服务>解决了只能使用普通用户来管理CM和Agent服务的需求.本文档 ...
CDH集群禁用kerberos的讲解
CDH集群禁用kerberos 情况: 我为什么会遇到这个需要禁用kerberos的情况,是因为我需要两个集群之前的数据迁移,源集群是有kerberos认证的,新集群也配置了认证,两个集群如果都配置了 ...
Hadoop集群搭 Hadoop分布式文件系统架构和设计
Hadoop集群搭建先决条件确保在你集群中的每个节点上都安装了所有必需软件. 获取Hadoop软件包. 安装安装Hadoop集群通常要将安装软件解压到集群内的所有机器上. 通常,集群里的一台机器 ...

CDH集群下hadoop异常PriviledgedActionException as:root (auth:SIMPLE)解决

CDH集群下hadoop异常PriviledgedActionException as:root (auth:SIMPLE)解决相关推荐

最新文章

热门文章