问题描述

Spark Streaming 启动读取Kafka时一直在报错，信息如下：

19/02/20 12:04:45 ERROR JobScheduler: Error running job streaming job 1550635380000 ms.0
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 10, slave23): org.apache.spark.SparkException: Couldn’t connect to leader for topic dsp_request_event 1: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator$$anonfun$connectLeader$1.apply(KafkaRDD.scala:164)
at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator$$anonfun$connectLeader$1.apply(KafkaRDD.scala:164)
at scala.util.Either.fold(Either.scala:97)
at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.connectLeader(KafkaRDD.scala:163)
at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.(KafkaRDD.scala:155)
at org.apache.spark.streaming.kafka.KafkaRDD.compute(KafkaRDD.scala:135)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

加粗的为定位的问题点，查看原始kafka的日志发现如下错误：

2019-02-20 10:52:33,371 INFO kafka.network.Acceptor: Rejected connection from /xx.xx.xx.xx, address already has the configured maximum of 10 connections.
2019-02-20 10:52:33,374 INFO kafka.network.Acceptor: Rejected connection from /xx.xx.xx.xx, address already has the configured maximum of 10 connections.
2019-02-20 10:52:52,186 INFO kafka.network.Acceptor: Rejected connection from /xx.xx.xx.xx, address already has the configured maximum of 10 connections.
2019-02-20 10:52:52,188 INFO kafka.network.Acceptor: Rejected connection from /xx.xx.xx.xx, address already has the configured maximum of 10 connections.

发现连接数错误，最终定位连接数问题。

解决方法

1、增加服务器的相关内核连接参数，发现并没有解决问题。
2、将flume连接kakfa的配置减少问题机器的连接时发现错误确实减少了，可以跑一段时间Streaming，但过一段时间后又失败了。
3、发现kafka有一个参数max.connections.per.ip即连接数设置为10，定位到问题，解决。

Rejected connection from xx.xx.xx.xx, address already has the configured maximum of 10 connections相关推荐

http://www.od85c.com.cn/html/,OllyDbg script for unpacking Enigma 4.xx and 5.xx
[Asm] 纯文本查看复制代码// Enigma Protector 4.xx and 5.XX unpacker by GIV (some parts are from LCF-AT Altern ...
java newsize_JVM中的-Xms -Xmx -XX:newSize -XX:MaxnewSize -Xmn -XX:PermSize -XX:MaxPermSize区别介绍...
一.配置参数使用条件在不断重构代码的情况下,仍然不能解决内存溢出的问题(OutOfMemoryError等),这种情况下就需要使用JVM的配置参数对JVM运行时各个区域的内存情况进行相应的分配. 二 ...
-XX:+UseParallelGC与 -XX:+UseParNewGC相关介绍
首先看一个图(所有的垃圾回收器,以及工作的堆空间) 对于我们默认(jdk1.7)使用parallel 垃圾回收器,old区就是使用的是parallel old, yong区使用的是parallel S ...
java xms512m_安装版的tomcat设置JAVA_OPTS=-Xms128M -Xmx512M -XX:PermSize=128M -XX:MaxPermSize=512M...
tomcat如果是通过windows服务启动,执行的是bin\tomcat.exe.他读取注册表中的值,而不是catalina.bat的设置.解决办法:修改注册表HKEY_LOCAL_MACHINE\ ...
【Java】计算从你的出生日期到现在相隔了多少天，多少时，XX分，XX秒。
计算从你的出生日期到现在相隔了多少天,多少时,XX分,XX秒.(例如:"XX学号为XXXXX,出生日期为:XX年XX月XX日,现在时间为:XX年XX月XX日,出生日期到现在相隔了多XX天,X ...
xx.h和xx.c的奥妙
xx.h和xx.c的奥妙示例代码: /********************* A.h*********************/ #ifndef __A_H__ #define __A_H__ ...
x+=x-=x*x; 和 x-=x*x; x+=x;
2019独角兽企业重金招聘Python工程师标准>>> 原问题: 第一种情况: int x=6; 问执行下列语句后x的值 x+=x-=x*x; 这个答案是-24. 首先明确一点赋值操 ...
bash xx.sh与sh xx.sh以及./xx.sh的区别
在ubuntu中,sh只是bash的一个链接,执行的程序都是一样的,命令解释器是一样的.但bash xx.sh和./xx.sh和sh xx.sh还是有不同的,bash xx.sh和sh xxx.sh都 ...
XX.sh和./XX.sh
./xx.sh执行当前目录中的xx.sh文件 xx.sh,执行名为xx.sh的文件后者如果在运行之前把.(指当前目录)加入到了环境变量中,且有可执行权限,则可以执行,否则如果如果没有加入环境变量则提 ...
当用户用input() 函数输入演员名字在屏幕上打印出xx出演电影xx
当用户用input() 函数输入演员名字在屏幕上打印出xx出演电影xx #创建一个电影库 dict1 = {"西游记": ["孙悟空", "猪八戒& ...

Rejected connection from xx.xx.xx.xx, address already has the configured maximum of 10 connections

Rejected connection from xx.xx.xx.xx, address already has the configured maximum of 10 connections.

问题描述

解决方法

Rejected connection from xx.xx.xx.xx, address already has the configured maximum of 10 connections相关推荐

最新文章

热门文章