KafkaConsumer is not safe for multi-threaded access

1、关于structed streaming 读取kafka的问题：

数据流从kafka过来，根据条件的不同，会被拆分成好几个流式dataFrame，这些流式的DF会进行各种join操作，在这个过程中会随机的触发KafkaConsumer is not safe for multi-threaded access，kafka多线程消费的问题，本程序就只有一个线程，怎么会有多线程消费的问题。在网上各方查找，有说是spark的架构task任务问题，有说是加个cache进行RDD缓存，有说是spark的bug，最终还是没有解决，问题还是会随机的出现，并且随着流的拆分数量增多，出现的概率也越高。（spark版本是2.3.0）

java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded accessat org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1431)at org.apache.kafka.clients.consumer.KafkaConsumer.seek(KafkaConsumer.java:1132)at org.apache.spark.sql.kafka010.CachedKafkaConsumer.seek(CachedKafkaConsumer.scala:305)at org.apache.spark.sql.kafka010.CachedKafkaConsumer.org$apache$spark$sql$kafka010$CachedKafkaConsumer$$fetchData(CachedKafkaConsumer.scala:216)at org.apache.spark.sql.kafka010.CachedKafkaConsumer$$anonfun$get$1.apply(CachedKafkaConsumer.scala:122)at org.apache.spark.sql.kafka010.CachedKafkaConsumer$$anonfun$get$1.apply(CachedKafkaConsumer.scala:106)at org.apache.spark.util.UninterruptibleThread.runUninterruptibly(UninterruptibleThread.scala:77)
Caused by: java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded accessat org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1431)at org.apache.kafka.clients.consumer.KafkaConsumer.seek(KafkaConsumer.java:1132)at org.apache.spark.sql.kafka010.CachedKafkaConsumer.seek(CachedKafkaConsumer.scala:305)at org.apache.spark.sql.kafka010.CachedKafkaConsumer.org$apache$spark$sql$kafka010$CachedKafkaConsumer$$fetchData(CachedKafkaConsumer.scala:216)at org.apache.spark.sql.kafka010.CachedKafkaConsumer$$anonfun$get$1.apply(CachedKafkaConsumer.scala:122)

2、解决办法：

由于问题是随机出现的，说明并不是代码的问题，可能是环境的问题或者其他外在的问题。目前找到唯一有效的办法就是提高spark的版本。将2.3.0改成了2.3.1。也不是最新版本，也不算老的版本，可能就是最好的版本。

<dependency><groupId>org.apache.spark</groupId><artifactId>spark-streaming_2.11</artifactId><version>2.3.1</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-hive_2.11</artifactId><version>2.3.1</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-streaming-kafka-0-10_2.11</artifactId><version>2.3.1</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-sql-kafka-0-10_2.11</artifactId><version>2.3.1</version></dependency>

KafkaConsumer is not safe for multi-threaded access相关推荐

【Kafka】KafkaConsumer is not safe for multi-threaded access
1.背景我写了一个消费kafka的程序报错这个. KafkaConsumer is not safe for multi-threaded accessException in thread &qu ...
spark streaming 整合kafka 报错 KafkaConsumer is not safe for multi-threaded access
问题描述 spark streaming 使用直连方式读取kafka 数据,使用窗口时出现 java.util.ConcurrentModificationException: KafkaCons ...
Kafka consumer多线程下not safe for multi-threaded access问题
Kafka consumer多线程下not safe for multi-threaded access问题默认配置下kafka consumer的offset的commit是自动的,如需改成手动提 ...
KafkaConsumer源码翻译(中英对照)
阅读kakka源码时,英语不好,翻译的比较费劲,经常翻译了后面的,忘记了前面说的什么了,等回到前面之后,前面说的是什么,英文又不认识了,在这里做一个记录,慢慢写吧,估计会写的很艰难,目前更新的不多,哎 ...
【MQ】Kafka笔记
笔记来源:尚硅谷视频笔记2.0版+2.1版黑马视频:Kafka深入探秘者来了 kafka笔记地址:https://blog.csdn.net/hancoder/article/details/107 ...
onload--Parameter Reference
Parameter List The parameter list details the following: •The environment variable used to set the p ...
vvvvvvvvvvvvvvvvvvvvvvvvv
Java Concurrency In Practice Brian Göetz Tim Peierls Joshua Bloch Joseph Bowbeer David Holmes Doug L ...
计算机科学与技术专业术语
计算机科学与技术专业术语 1.CPU 3DNow!(3D no waiting) ALU(Arithmetic Logic Unit,算术逻辑单元) AGU(Address Generation Un ...
Kafka的常见错误分析
1.UnknownTopicOrPartitionException org.apache.kafka.common.errors.UnknownTopicOrPartitionException: ...

KafkaConsumer is not safe for multi-threaded access

KafkaConsumer is not safe for multi-threaded access相关推荐

最新文章

热门文章