Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

CDH5.16.2执行spark-submit或者spark-shell 时报错

[root@hadoop103 ~]# spark-submit
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStreamat org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:123)at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:123)at scala.Option.getOrElse(Option.scala:120)at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:123)at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:109)at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:114)at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStreamat java.net.URLClassLoader.findClass(URLClassLoader.java:382)at java.lang.ClassLoader.loadClass(ClassLoader.java:424)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)at java.lang.ClassLoader.loadClass(ClassLoader.java:357)... 7 more

原因是CDH版的Spark从1.4版本以后，Spark编译时都没有将hadoop的classpath编译进去，所以必须在spark-env.sh中指定hadoop中的所有jar包。

解决方案：

在spark-env.sh中添加一条配置信息，将hadoop的classpath引入, ${HADOOP_HOME}根据自己的情况而定，直接写绝对路径也行；注意所有节点都需要修改。

export SPARK_DIST_CLASSPATH=$(${HADOOP_HOME}/bin/hadoop classpath)

CM：

去CM网页上修改Spark2.2配置，指定SPARK_DIST_CLASSPATH，然后重启过期配置。

export SPARK_DIST_CLASSPATH=$(${HADOOP_HOME}/bin/hadoop classpath)

特别注意：CDH升级Spark2后，提交命令是spark2-submit

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream相关推荐

集群提交HBase代码报错：Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguratio
原因分析 HBase官网已经告诉我们原因以及解决方法了: 原因: 默认情况下,部署到MapReduce群集的MapReduce作业无法访问$ HBASE_CONF_DIR下的HBase配置或HBas ...
Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
spark-env.sh中加入 export SPARK_DIST_CLASSPATH=$(${HADOOP_HOME}/bin/hadoop classpath)
flink启动出现java.lang.NoClassDefFoundError: org/apache/hadoop/fs/Path
flink的版本是1.10.1 yarn-session.sh启动后出现如下报错: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/Path ...
IDEA里运行代码时出现Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Logger的解决办法（图文详解）...
问题详情运行出现log4j的问题 "C:\Program Files\Java\jdk1.8.0_66\bin\java" -Didea.launcher.port=7533 & ...
Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
1.错误叙述性说明 2014-7-12 0:38:57 org.apache.catalina.core.ApplicationContext log 信息: No Spring WebApplica ...
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Priority 的解决办法
问题概述: 使用 Spring Boot-2.0.3 进行项目开发,在配置 Druid 数据监控时,遇到缺包的问题,测试连接操作时报错,导致测试一直无法通过,部分信息如下: Caused by: ja ...
解决Mybatis Plus代码自动生成时报错： Caused by: java.lang.ClassNotFoundException: org.apache.velocity.context
解决Mybatis Plus代码自动生成时报错: Caused by: java.lang.ClassNotFoundException: org.apache.velocity.context.Co ...
intellij中出現org.apache.hadoop.fs.FSDataInputStream解決方案
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream 在intellij中的办法是导入 ...
java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException
flink提交任务卡死,cancel job以后,在$FLINK_HOME/log/flink-appleyuchi-client-Desktop.log 发现该报错: java.lang.NoCla ...

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream相关推荐

最新文章

热门文章