提交spark任务偶尔报错 org.apache.spark.SparkException: A master URL must be set in your configuration

问题一：pox.xml里外部以来库没有加入<scope>compile</scope>导致无法初始化

问题二：main函数以外的方法有时也无法初始化

错误信息：

20/11/24 22:32:00 INFO DAGScheduler: ResultStage 0 (take at Base64UserEmb.scala:57) failed in 16.761 s due to Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 9.10.8.90, executor 3): java.lang.ExceptionInInitializerErrorat com.tencent.ieg.dm.pltv.feature.Base64UserEmb$$anonfun$3.apply(Base64UserEmb.scala:52)at com.tencent.ieg.dm.pltv.feature.Base64UserEmb$$anonfun$3.apply(Base64UserEmb.scala:48)at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)at scala.collection.Iterator$$anon$10.next(Iterator.scala:394)at scala.collection.Iterator$class.foreach(Iterator.scala:891)at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)at scala.collection.AbstractIterator.to(Iterator.scala:1334)at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334)at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)at scala.collection.AbstractIterator.toArray(Iterator.scala:1334)at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$31.apply(RDD.scala:1409)at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$31.apply(RDD.scala:1409)at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2162)at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2162)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)at org.apache.spark.scheduler.Task.run(Task.scala:123)at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1419)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: A master URL must be set in your configurationat org.apache.spark.SparkContext.<init>(SparkContext.scala:368)at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2581)at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:976)at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:967)at scala.Option.getOrElse(Option.scala:121)at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:967)at com.tencent.ieg.dm.utils.RunUtil$.createCluterContext(RunUtil.scala:52)at com.tencent.ieg.dm.utils.TDWApp$class.$init$(RunUtil.scala:23)at com.tencent.ieg.dm.pltv.feature.Base64UserEmb$.<init>(Base64UserEmb.scala:11)at com.tencent.ieg.dm.pltv.feature.Base64UserEmb$.<clinit>(Base64UserEmb.scala)... 27 more

红色的是关键信息，最后找到了这篇文章：https://www.cnblogs.com/barneywill/p/10109122.html 解决了大问题！

提交spark任务偶尔报错 org.apache.spark.SparkException: A master URL must be set in your configuration相关推荐

Spark-submit 提交报错 org.apache.spark.sql.execution.datasources.orc.OrcFileFormat could not be instant
错误场景如下代码: spark.sql("select e.empno,e.ename,e.job,e.mgr,e.comm from emp e join dept d on e.dep ...
spark学习：org.apache.spark.SparkException: A master URL must be set in your config
Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in yo ...
spark 序列化错误集群提交时_【问题解决】本地提交任务到Spark集群报错：Initial job has not accepted any resources...
本地提交任务到Spark集群报错:Initial job has not accepted any resources 错误信息如下: 18/04/17 18:18:14 INFO TaskSched ...
【Flink】Flink 提交任务到yarn报错 proxy provider ConfiguredFailoverProxyProvider NetUtils.getSocketAddressS
文章目录 1.概述 1.概述 Flink 提交任务到yarn报错 Couldn't create proxy provider class org.apache.hadoop.hdfs.server. ...
Spark SQL入门：创建SparkSession时import spark.implicits._ 报错： error: value implicits is not a member of...
Spark SQL入门:创建SparkSession时import spark.implicits._ 报错: error: value implicits is not a member of... ...
mongodb偶尔报错com.mongodb.MongoSocketReadException: Prematurely reached end of stream
项目开发中,链接mongodb的项目,偶尔报错com.mongodb.MongoSocketReadException: Prematurely reached end of stream 报错的详细 ...
spring boot一个模块加载不到引用另一个模块的mapper.xml报错org.apache.ibatis.binding.BindingException: Invalid bound sta
场景:parent项目有两个子模块,分别是shiro和server,两个子模块各自有各自的实体类.mapper,然后server需要引用shiro中的实体类和mapper.已经在启动类添加注解配置扫描 ...
@webservice报错org.apache.cxf.common.i18n.UncheckedException: No operation was found with
文章目录 1. 现象 2. 解决办法1 3. 解决办法2 1. 现象整合spring+cxf的webservice,成功发布了wsdl,但在调用的时候报错 org.apache.cxf.common ...
SpringBoot报错 org.apache.catalina.LifecycleException: Protocol handler start failed
很多人在第一次创建运行SpringBoot项目的时候会报错 org.apache.catalina.LifecycleException: Protocol handler start failed ...

提交spark任务偶尔报错 org.apache.spark.SparkException: A master URL must be set in your configuration

提交spark任务偶尔报错 org.apache.spark.SparkException: A master URL must be set in your configuration相关推荐

最新文章

热门文章