IDEA 本地调试spark程序 Exception in thread main java.lang.NoSuchMethodError: scala.collection.immutable.

1.很简单的一测试程序：

2.run运行时：出现ERROR,报错行显示在 new SparkContext 这行。

val sc = new SparkContext(conf)

Exception in thread "main" java.lang.NoSuchMethodError: scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet;
   at akka.actor.ActorCell$.<init>(ActorCell.scala:336)
   at akka.actor.ActorCell$.<clinit>(ActorCell.scala)
   at akka.actor.RootActorPath.$div(ActorPath.scala:159)
   at akka.actor.LocalActorRefProvider.<init>(ActorRefProvider.scala:464)
   at akka.remote.RemoteActorRefProvider.<init>(RemoteActorRefProvider.scala:124)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78)
   at scala.util.Try$.apply(Try.scala:192)
   at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73)
   at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
   at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
   at scala.util.Success.flatMap(Try.scala:231)
   at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84)
   at akka.actor.ActorSystemImpl.liftedTree1$1(ActorSystem.scala:584)
   at akka.actor.ActorSystemImpl.<init>(ActorSystem.scala:577)
   at akka.actor.ActorSystem$.apply(ActorSystem.scala:141)
   at akka.actor.ActorSystem$.apply(ActorSystem.scala:118)
   at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:122)
   at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:54)
   at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53)
   at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1991)
   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
   at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1982)
   at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:56)
   at org.apache.spark.rpc.akka.AkkaRpcEnvFactory.create(AkkaRpcEnv.scala:245)
   at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:52)
   at org.apache.spark.SparkEnv$.create(SparkEnv.scala:247)
   at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:188)
   at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:267)
   at org.apache.spark.SparkContext.<init>(SparkContext.scala:424)
   at com.spark.sample.WordCount$.main(WordCount.scala:11)
   at com.spark.sample.WordCount.main(WordCount.scala)
20/03/11 21:53:22 INFO Utils: Shutdown hook called
3.百度，码友说scala 和spark 版本不兼容导致。

本地安装的scala 版本是：2.11.12

idea 上Global Libraries 版本也是：2.11.12

由于saprk程序是maven项目，查看pom

<dependency><groupId>org.apache.spark</groupId><artifactId>spark-core_2.10</artifactId><version>1.4.0</version>
</dependency>

4.到 maven repository 官网，查找scala匹配的spark版本，更改为：

<dependency><groupId>org.apache.spark</groupId><artifactId>spark-core_2.11</artifactId><version>2.1.0</version>
</dependency>

5.重试程序，OK.通过。

在此感谢码友！！！

IDEA 本地调试spark程序 Exception in thread main java.lang.NoSuchMethodError: scala.collection.immutable.相关推荐

Spark启动错误Exception in thread “main” java.lang.NoSuchMethodError: scala.collection.immutable.HashSet$
1 错误 [error] (run-main-0) java.lang.NoSuchMethodError: scala.collection.immutable.HashSet$.empty() ...
Exception in thread main java.lang.NoSuchMethodError: scala.collection.immutable.HashSet$.empty()L
Scala 的版本与 Spark版本不匹配博主用的是 Spark 1.5.1 之前装的 Scala 2.11.0 ,运行示例SparkPi出现上述错误. 改用Scala 2.10.6 ,问题解决了.
Exception in thread “main“ java.lang.NoSuchMethodError: scala.collection.mut
scala版本不一致,采用maven中spark自带版本即可.
Exception in thread “main“ java.lang.NoSuchMethodError: scala.reflect.ClassT
运行环境:IntelliJ + scala + spark +hadoop 出错原因:IntelliJ 中的scala版本与Spark 自带的scala版本不同导致我刚开始以为只要把IntelliJ ...
Spark 启动报错时报错 java.lang.NoSuchMethodError: scala.collection.immutable.HashSet$.empty()Lscala/collect
今天在Spark测试时异常报错一直提示找不到该方法因为本身代码量很小而且没有用到错误提示的类找了许久才发现是版本对应问题 /Library/Java/JavaVirtu ...
Exception in thread “main“ java.lang.NoSuchMethodError: scala.Predef$
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$ (我这里是idea + spark + ...
Exception in thread main java.lang.NoSuchMethodError: scala.Predef$
使用intelli idea +scala+spark,运行程序代码如下: package cn.limbo.sparkimport org.apache.spark.SparkConf import ...
Exception in thread main java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)
其实这个错误就是编译的程序的scala版本和提交时的版本不一致导致的. 我是在idea上写的scala程序用的是2.10.5, 但是在提交spark时忽然发现spark用的是2.11.12 所 ...
Exception in thread main java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less
网上有两种说法:依赖冲突(conflicts of dependencies)以及多版本的Scala(likely have multiple major versions of scala that ...

IDEA 本地调试spark程序 Exception in thread main java.lang.NoSuchMethodError: scala.collection.immutable.

IDEA 本地调试spark程序 Exception in thread main java.lang.NoSuchMethodError: scala.collection.immutable.相关推荐

最新文章

热门文章