1.现象

spark程序报错后重新启动后报警

22/05/05 10:26:54 ERROR executor.Executor: Exception in task 2.1 in stage 3.0 (TID 37)
org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: {eventshistory-0=16914080}at org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:987)at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:490)at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1256)at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1188)at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1123)at org.apache.spark.streaming.kafka010.InternalKafkaConsumer.poll(KafkaDataConsumer.scala:200)at org.apache.spark.streaming.kafka010.InternalKafkaConsumer.get(KafkaDataConsumer.scala:129)at org.apache.spark.streaming.kafka010.KafkaDataConsumer$class.get(KafkaDataConsumer.scala:36)at org.apache.spark.streaming.kafka010.KafkaDataConsumer$NonCachedKafkaDataConsumer.get(KafkaDataConsumer.scala:218)at org.apache.spark.streaming.kafka010.KafkaRDDIterator.next(KafkaRDD.scala:261)at org.apache.spark.streaming.kafka010.KafkaRDDIterator.next(KafkaRDD.scala:229)at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)at scala.collection.Iterator$class.foreach(Iterator.scala:891)at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)at com.tcl.realtime.app.AppEventHistory$$anonfun$main$1$$anonfun$2.apply(AppEventHistory.scala:94)at com.tcl.realtime.app.AppEventHistory$$anonfun$main$1$$anonfun$2.apply(AppEventHistory.scala:74)at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2121)at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2121)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)at org.apache.spark.scheduler.Task.run(Task.scala:121)at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1408)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)

2.报错原因

log.retention.ms

kafka的日志保存时间设置了是3天，因为程序报错中间没有运行，从新启动运行的时候从之前的地方开始消费的时候日志已经被kafka清空了。因此报警

3.报错解决

将spark从之前消费的记录删除。从最新的地方开始消费。相同数据下游只会入库一次。因此将未消费的数据重新采集到kafka对应主题中。

org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured相关推荐

OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions
问题描述: OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions ...
java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord
java消费kafka数据时报错 ERROR [Executor task launch worker for task 90] - Exception in task 0.0 in stage 54 ...
【Flink实战系列】Lorg/apache/flink/kafka/shaded/org/apache/kafka/clients/consumer/ConsumerRecord；)Ljava/
java.lang.AbstractMethodError: Method flink/stream/deserialization/PoJoDeserializationSchema.deseria ...
【记一次kafka报org.apache.kafka.clients.consumer.CommitFailedException异常处理】
项目场景: 项目中,使用到了kafka作为消息中间件,项目作为消费端,消费消息并进行业务处理问题描述在实际应用的过程中,发现偶尔但是一直存在的,有消费数据报:org.apache.kafka.cl ...
java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.assign(Ljava/util/List
版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明. 本文链接: Flink入门程序异常,记录一下跟大家分享. SLF4J: Failed to l ...
spark2+kafka报错:java.lang.NoSuchMethodError:org.apache.kafka.clients.consumer.KafkaConsumer.subscribe
spark读取kafka数据 // Create DataFrame representing the stream of input lines from kafkaval lines = spar ...
kafka生产者报错：[org.apache.kafka.clients.NetworkClient:600] - Error while fetching metadata with corre
在测试kafka写入数据的时候一直在报错: ... [org.apache.kafka.clients.NetworkClient:600] - Error while fetching meta ...
Error:(3, 41) java: 程序包org.apache.kafka.clients.producer不存在错误提示解决办法
解决 mvn idea:module
Apache Kafka Consumer 消费者集
1.目标在我们的上一篇文章中,我们讨论了Kafka Producer.今天,我们将讨论Kafka Consumer.首先,我们将看到什么是Kafka Consumer和Kafka Consumer的 ...
第三集 Spring for Apache Kafka 接受消息
我们可以接受消息通过配置一个MessageListenerContainer 和提供一个消息监听或者通过使用@KafkaListener 注解 3.1 Message Listeners 当我们使用一 ...

org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured

1.现象

2.报错原因

3.报错解决

org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured相关推荐

最新文章

热门文章