java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord

java消费kafka数据时报错

ERROR [Executor task launch worker for task 90] - Exception in task 0.0 in stage 54.0 (TID 90)
java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord
Serialization stack:- object not serializable (class: org.apache.kafka.clients.consumer.ConsumerRecord, value: ConsumerRecord(topic = cctv, partition = 2, offset = 483, CreateTime = -1, checksum = 2620948080, serialized key size = 1, serialized value size = 1704, key = 0, value = 15/Dec/2020:10:17:42 +0800#CS#POST /B2C40/query/jaxb/direct/query.ao HTTP/1.1#CS#POST#CS#application/x-www-form-urlencoded; charset=UTF-8#CS#json=%7B%22depcity%22%3A%22CAN%22%2C+%22arrcity%22%3A%22WUH%22%2C+%22flightdate%22%3A%2220180220%22%2C+%22adultnum%22%3A%221%22%2C+%22childnum%22%3A%220%22%2C+%22infantnum%22%3A%220%22%2C+%22cabinorder%22%3A%220%22%2C+%22airline%22%3A%221%22%2C+%22flytype%22%3A%220%22%2C+%22international%22%3A%220%22%2C+%22action%22%3A%220%22%2C+%22segtype%22%3A%221%22%2C+%22cache%22%3A%220%22%2C+%22preUrl%22%3A%22%22%2C+%22isMember%22%3A%22%22%7D#CS#http://b2c.csair.com/B2C40/modules/bookingnew/main/flightSelectDirect.html?t=S&c1=CAN&c2=WUH&d1=2020-12-15&at=1&ct=0&it=0#CS#192.168.10.1#CS#Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36#CS#2020-12-15T10:17:42+08:00#CS#192.168.10.101#CS#JSESSIONID=782121159357898886-112554CF44321E; sid=b5cc11e02e154ac5b0f3609332f86803; aid=8ae8768760927e280160bb348bef3e12; identifyStatus=N; userType4logCookie=M; userId4logCookie=13818791413; useridCookie=13818791413; userCodeCookie=13818791413; temp_zh=cou%3D0%3Bsegt%3D%E5%8D%95%E7%A8%8B%3Btime%3D2018-01-13%3B%E5%B9%BF%E5%B7%9E-%E5%8C%97%E4%BA%AC%3B1%2C0%2C0%3B%26cou%3D1%3Bsegt%3D%E5%8D%95%E7%A8%8B%3Btime%3D2020-12-15%3B%E5%B9%BF%E5%B7%9E-%E6%88%90%E9%83%BD%3B1%2C0%2C0%3B%26; JSESSIONID=782121159357898886-112554CF44321E; WT-FPC=id=211.103.142.26-608782688.30635197:lv=1516170718655:ss=1516170709449:fs=1513243317440:pn=2:vn=10; language=zh_CN; WT.al_flight=WT.al_hctype(S)%3AWT.al_adultnum(1)%3AWT.al_childnum(0)%3AWT.al_infantnum(0)%3AWT.al_orgcity1(CAN)%3AWT.al_dstcity1(CTU)%3AWT.al_orgdate1(2020-12-15)#CS#1))- element of array (index: 0)- array (class [Lorg.apache.kafka.clients.consumer.ConsumerRecord;, size 11)at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46)at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:392)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)
ERROR [task-result-getter-1] - Task 0.0 in stage 54.0 (TID 90) had a not serializable result: org.apache.kafka.clients.consumer.ConsumerRecord
Serialization stack:

解决方法

设置kryo序列化

val spark: SparkSession = SparkSession.builder().appName("StreamingTest").master("local[6]").config("spark.serializer", "org.apache.spark.serializer.KryoSerializer").getOrCreate()

java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord相关推荐

java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.assign(Ljava/util/List
版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明. 本文链接: Flink入门程序异常,记录一下跟大家分享. SLF4J: Failed to l ...
spark2+kafka报错:java.lang.NoSuchMethodError:org.apache.kafka.clients.consumer.KafkaConsumer.subscribe
spark读取kafka数据 // Create DataFrame representing the stream of input lines from kafkaval lines = spar ...
【Flink实战系列】Lorg/apache/flink/kafka/shaded/org/apache/kafka/clients/consumer/ConsumerRecord；)Ljava/
java.lang.AbstractMethodError: Method flink/stream/deserialization/PoJoDeserializationSchema.deseria ...
【记一次kafka报org.apache.kafka.clients.consumer.CommitFailedException异常处理】
项目场景: 项目中,使用到了kafka作为消息中间件,项目作为消费端,消费消息并进行业务处理问题描述在实际应用的过程中,发现偶尔但是一直存在的,有消费数据报:org.apache.kafka.cl ...
java.io.NotSerializableException: org.apache.shiro.util.SimpleByteSource at java.io.ObjectOutputStr
异常背景: 做shiro+salt+redis整合时,一直报不能序列化错误,不开启redis时还是正常的,开启redis后就出错. 问题描述: 自定义CustomerRealm extends Aut ...
Error:(3, 41) java: 程序包org.apache.kafka.clients.producer不存在错误提示解决办法
解决 mvn idea:module
java序列化错在哪里_Spark序列化错误：java.io.NotSerializableException
由于spark算子用到的class没有实现序列化,报错如下所示 15/11/23 14:43:47 ERROR Executor: Exception in task 0.0 in stage 4.0 ...
Mybatis中的java.io.NotSerializableException
在学习Mybatis中,用JUnit运行了一个测试方法,出现了如下错误: org.apache.ibatis.cache.CacheException: Error serializing objec ...
Caused by: java.io.NotSerializableException:cn.bloghut.pojo.User
错误产生环境 mybatis 项目开启二级缓存时产生错误描述 Caused by: java.io.NotSerializableException: cn.bloghut.pojo.Userat ...

java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord

解决方法

java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord相关推荐

最新文章

热门文章