java.io.IOException: No FileSystem for scheme: hdfs转载加自己笔记

问题出现使用maven+scala+spark的时候,运行
java -jar xxx.jar的时候出现的:

java.io.IOException: No FileSystem for scheme: hdfsat org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)at org.apache.spark.deploy.yarn.Client$$anonfun$5.apply(Client.scala:123)at org.apache.spark.deploy.yarn.Client$$anonfun$5.apply(Client.scala:123)at scala.Option.getOrElse(Option.scala:121)at org.apache.spark.deploy.yarn.Client.<init>(Client.scala:123)at org.apache.spark.deploy.yarn.Client.<init>(Client.scala:69)at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:55)at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)

解决方案1：

    val spark = SparkSession.builder.appName("Intro").config("spark.master", "local").getOrCreate();spark.sparkContext.setLogLevel("ERROR")val hadoopConf = spark.sparkContext.hadoopConfigurationhadoopConf.set("fs.hdfs.impl", classOf[org.apache.hadoop.hdfs.DistributedFileSystem].getName)hadoopConf.set("fs.file.impl", classOf[org.apache.hadoop.fs.LocalFileSystem].getName)

解决方案2：
在core-site.xml中加入如下配置，并且重启，以后每次都会生效。

<property><name>fs.file.impl</name><value>org.apache.hadoop.fs.LocalFileSystem</value>
</property>
<property><name>fs.hdfs.impl</name><value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
</property>

java.io.IOException: No FileSystem for scheme: hdfs转载加自己笔记相关推荐

java.io.IOException: No FileSystem for scheme: hdfs
转自:http://www.cnblogs.com/justinzhang/p/4983673.html 介绍了如何将Maven依赖的包一起打包进jar包.使用maven-assembly打成jar后 ...
java.io.IOException No FileSystem for scheme hdfs
java.io.IOException: No FileSystem for scheme: hdfs 以下将转载两篇博文,博文中有解决方法. 1.Java下Maven打包项目解决方法 log4j:W ...
【hadoop】java.io.IOException: No FileSystem for scheme: hdfs
文章目录 1.概述 1.概述异常 org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme &quo ...
Exception in thread main java.io.IOException: No FileSystem for scheme: hdfs
如果是命令行spark-submit运行时出现这个问题,那么: import org.apache.spark.sql.{Dataset, SparkSession}val spark = Spark ...
Eclipse运行Spark程序报错No FileSystem for scheme: hdfs
场景:通过spark程序读取hdfs文件. 错误: Exception in thread "main" java.io.IOException: No FileSystem fo ...
hadoop 修复hdfs中文件块错误和文件修复 Error: java.io.IOException: Cannot obtain block length for LocatedBlock
一:问题 Error: java.io.IOException: Cannot obtain block length for LocatedBlock Error: java.io.IOExcept ...
PriviledgedActionException as:Administrator cause:java.io.IOException
在windows下用Eclipse运行MapReduce程序会报错: 12/04/24 15:32:44 WARN util.NativeCodeLoader: Unable to load nati ...
linux rm 提示io异常,Hadoop异常 java.io.IOException: Job status not available
Hadoop集群上跑mapreduce,在job任务执行完成退出时报 java.io.IOException: Job status not available异常.Job client请求job状态 ...
java.io.IOException: Incompatible clusterIDs
Datanode未成功启动,查看slave节点的日志,发现报如下错误java.io.IOException: Incompatible clusterIDs. 原因如下: 执行hdfs namenod ...

java.io.IOException: No FileSystem for scheme: hdfs转载加自己笔记

java.io.IOException: No FileSystem for scheme: hdfs转载加自己笔记相关推荐

最新文章

热门文章