java序列化错在哪里_Spark序列化错误:java.io.NotSerializableException
由于spark算子用到的class没有实现序列化,报错如下所示
15/11/23 14:43:47 ERROR Executor: Exception in task 0.0 in stage 4.0 (TID 4)
java.io.NotSerializableException: EntityMention
Serialization stack:
- object not serializable (class: EntityMention, value: EntityMention@5cdadff6)
at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
at org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:147)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:181)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.insertAll(BypassMergeSortShuffleWriter.java:121)
at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
15/11/23 14:43:47 ERROR TaskSetManager: Task 0.0 in stage 4.0 (TID 4) had a not serializable result: EntityMention
Serialization stack:
- object not serializable (class: EntityMention, value: EntityMention@5cdadff6); not retrying
15/11/23 14:43:47 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
15/11/23 14:43:47 INFO TaskSchedulerImpl: Cancelling stage 4
15/11/23 14:43:47 INFO DAGScheduler: ShuffleMapStage 4 (groupBy at Relation_Detector.scala:78) failed in 0.081 s
15/11/23 14:43:47 INFO DAGScheduler: Job 4 failed: collect at Relation_Detector.scala:79, took 0.097801 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0 in stage 4.0 (TID 4) had a not serializable result: EntityMention
Serialization stack:
- object not serializable (class: EntityMention, value: EntityMention@5cdadff6)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1822)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1835)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1848)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1919)
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:905)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
at org.apache.spark.rdd.RDD.collect(RDD.scala:904)
at Relation_Detector$.main(Relation_Detector.scala:79)
at Relation_Detector.main(Relation_Detector.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
15/11/23 14:43:47 INFO SparkContext: Invoking stop() from shutdown hook
解决方法:对类实现序列化Serializable
class EntityMention(val pentitymention_id:String,var pname:String,var pentitymention_type:String,
var pHbaseID:String,var ptopicID:String,var ppost_type:String,
var psenIndex:String,var psenOffset:String, var ppostTime:String,
var pinsertTime:String,var pentityID:String) extends Serializable {
var entitymention_id = pentitymention_id
var name = pname
var entitymention_type = pentitymention_type
var HbaseID = pHbaseID
var topicID= ptopicID
var post_type = ppost_type
var senIndex = psenOffset
var senOffset = psenOffset
var postTime = ppostTime
var insertTime = pinsertTime
var entityID = pentityID
}
java序列化错在哪里_Spark序列化错误:java.io.NotSerializableException相关推荐
- java.lang.arr_以太坊常见问题和错误 / java.lang.ArrayIndexOutOfBoundsException? - 汇智网...
web3j:智能合约写操作时总是抛出java.lang.ArrayIndexOutOfBoundsException错误,不知道什么问题. 我的代码如下: String hexPrivateKey = ...
- java url no protocol_httpurlconnection 新人使用遇到错误java.net.MalformedURLException: no protocol...
自己在琢磨中:publicclasshttpurl{ publicstaticvoidmain(String[]args) { try{URLurl=newURL("www.s ...
- 你了解java序列化吗?一篇文章搞懂java序列化
程序员的成长之路 互联网/程序员/技术/资料共享 关注 阅读本文大概需要 9 分钟. 来自:https://www.cnblogs.com/xiaohuiduan/p/12398775.html 首先 ...
- 序列化与反序列化(1)Serializable —— Java原生态方法
摘自:序列化与反序列化(1)Serializable -- Java原生态方法 作者:丶PURSUING 发布时间: 2021-05-08 19:20:21 网址:https://blog.csdn. ...
- java序列化和反序列化_Java恶意序列化背后的历史和动机
java序列化和反序列化 与Java的序列化机制相关的问题已广为人知. 有效的Java 1st Edition (第10章)和有效的Java 2nd Edition (第11章)的整个最后一章都专门讨 ...
- java继承序列化_java中序列化之子类继承父类序列化
当一个父类实现Serializable接口后,他的子类都将自动的实现序列化. 以下验证了这一点: package Serial; import java.io.Serializable; public ...
- Java对象表示方式1:序列化、反序列化和 transient 关键字的作用
前言快速到底 平时我们接触到的 Java 对象全都是在内存中的对象(它像磁盘中的一个个文件那样实实在在的存在着),所以它是无法进行 IO 操作或者网络通信的,因为在进行 IO 操作或者网络通信的时候, ...
- feign 序列化_Spring Boot和Feign中使用Java 8时间日期API(LocalDate等)的序列化问题...
LocalDate.LocalTime.LocalDateTime是Java 8开始提供的时间日期API,主要用来优化Java 8以前对于时间日期的处理操作.然而,我们在使用Spring Boot或使 ...
- 【JAVA SE】第十三章 序列化与反序列化
第十三章 序列化与反序列化 文章目录 第十三章 序列化与反序列化 一.概念 1.序列化与反序列化 2.序列化有什么作用 二.代码实现 1.思路 2.创建一个Student类 3.创建TestSeria ...
最新文章
- 学习进度条--第七周
- 常用的数据统计Sql 总结(转)
- matlab--离散(discrete)数据绘图
- 0116互联网新闻 | “DaDa英语”近日完成2.55亿美元D轮融资;腾讯推今年首个手游“闹闹天宫”...
- android开发将h5转换成pdf_一键将PDF转换成PPT,秒懂!
- 使用redis批量生成主键(订单)Id
- C语言版数据结构及算法_快速排序
- linux cp 强行覆盖
- 如何站在双11的肩膀上 详解阿里云企业级互联网架构
- sobel算子_边缘检测算法4.-教你动手实现kirsch和robinson算子
- ocp007 题库解析(1-15)
- linux微信电脑版登录不了,默认Windows与Deepin系统下的微信电脑版目录
- android 华为摄像头权限_Android踩坑日记(一):android7.0动态相机权限
- 高效人士的七个管理习惯
- Communication(Floyed+并查集 || Trajan)
- win10安装Visual Studio2019时卡在了提取文件处
- 2017知乎看山杯总结(多标签文本分类)
- Windows 10正式版官方原版ISO镜像下载汇总!!!
- Mtk Sensor 驱动框架变更简要分析
- CCF中学生计算机程序设计入门篇练习题讲解
热门文章
- 不间断电源ups标准_UPS不间断电源正确使用方法
- 11210怎么等于24_想要消耗100大卡热量,怎么做才最简单?
- vim关闭下一行也是注释_在Vim中注释/取消注释行的快速方法是什么?
- gradle compile mysql_Gradle配置implementation、api与compile的区别
- python分治算法_分治法及其python实现例子
- win10文件显示后缀名_Win10一开机,内存占用竟高达60%?你可以尝试这样做
- dw中html颜色的设置颜色代码,Dreamweaver 如何编辑字体大小颜色
- linux mei swap,Linux swapoff命令
- mysql架设_主从mysql架设
- Win7系统账户被禁用的解决方法