spark编程ERROR01——java.lang.NullPointerException

IDEA中运行spark程序，发现datafram可以进行df.show() 但是只要df.count() 就会显示下面的异常信息：

2022-03-25 17:56:13,691 ERROR executor.Executor: Exception in task 14.0 in stage 7.0 (TID 222)
java.lang.NullPointerExceptionat $line33.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.$anonfun$rdd01$1(<console>:26)at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.agg_doAggregateWithKeys_0$(Unknown Source)at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)at org.apache.spark.scheduler.Task.run(Task.scala:131)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)
2022-03-25 17:56:13,728 WARN scheduler.TaskSetManager: Lost task 14.0 in stage 7.0 (TID 222) (westgis-134 executor driver): java.lang.NullPointerExceptionat $line33.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.$anonfun$rdd01$1(<console>:26)at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.agg_doAggregateWithKeys_0$(Unknown Source)at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)at org.apache.spark.scheduler.Task.run(Task.scala:131)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)

之所以出现这种错误，是因为前面的处理中datafram属性里存在NULL值，用na.drop()去除，错误得到解决。
df05=df05.select("direction","station_name","order_no","lat","lng").na.drop()

spark编程ERROR01——java.lang.NullPointerException相关推荐

spark报错 java.lang.NullPointerException: Null value appeared in non-nullable field:top level Product
报错内容 java.lang.NullPointerException: Null value appeared in non-nullable field: top level Product in ...
【已解决】关于IDEA中 Driver 出现Exception in thread quot；mainquot； java.lang.NullPointerException问题
在学习hive过程中,按照mapreduce编程规范,分别编写Mapper,Reducer,Driver,在编写Driver后运行报错了,花了两天,找到了解决方案(裂开.JPG). 代码: packa ...
Null value appeared in non-nullable field java.lang.NullPointerException
报错 Null value appeared in non-nullable field java.lang.NullPointerException: Null value appeared in ...
java.lang.NullPointerException
java.lang.NullPointerException is one of the most popular exceptions in java programming. Anybody wo ...
java.lang.NullPointerException出现的几种原因及解决方案
java.lang.NullPointerException是java编程中最常见的异常之一.任何使用java的人都有在java程序以及java web应用程序中看到java.lang.NullPoi ...
java.lang.NullPointerException异常原因及解决
java.lang.NullPointerException异常原因是因为创建了一个引用类型的变量却没有指向任何对象而又去通过这个引用类型变量加点的形式去访问非静态的方法及属性. 给出三种情况, 第一 ...
java.lang.NullPointerException: Attempt to invoke virtual method ‘boolean java.lang.String.equals(j
今天切正式环境出现的空指针记录一下下面是错误 java.lang.NullPointerException: Attempt to invoke virtual method 'boolean j ...
绝望，上传文件失败。。遇到并解决java.lang.NullPointerException
解决报错:java.lang.NullPointerException 解决方法:将文件的编码方式改为utf-8即可! 截图: 修改成功后: 代码: import os import json imp ...
java.lang.NullPointerException空指针问题
对于一个Android新手而言遇见空指针问题是在所难免的.比如在调试时一开程序就出现"很抱歉!'XXX'已停止运行!"这是空指针的一种表现. 为什么会出现空指针? 所谓空指针异常, ...
解决非controller使用@Autowired注解注入报错为java.lang.NullPointerException问题
解决非controller使用@Autowired注解注入报错为java.lang.NullPointerException问题参考文章: (1)解决非controller使用@Autowired注 ...

spark编程ERROR01——java.lang.NullPointerException

spark编程ERROR01——java.lang.NullPointerException相关推荐

最新文章

热门文章