java.io.IOException: Connection reset by peer
spark通过phoenix 读取hbase表,其实说白了先要去Zookeeper建立connection。
二.代码:
点击(此处)折叠或打开
val formatStr = "org.apache.phoenix.spark"
val oms_orderinfoDF = spark.read.format(formatStr)
.options(Map("table" -> "oms_orderinfo", "zkUrl" -> zkUrl))
.load
三.查看SparkJob日志:
点击(此处)折叠或打开
17/10/24 03:25:25 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.100.48:35952, server: hadoop40/192.168.100.40:2181
17/10/24 03:25:25 WARN zookeeper.ClientCnxn: Session 0x0 for server hadoop40/192.168.100.40:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
17/10/24 03:25:25 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://nameservice1/user/hdfs/.sparkStaging/application_1507703377455_4854
17/10/24 03:25:25 INFO util.ShutdownHookManager: Shutdown hook called
主要错误是“java.io.IOException: Connection reset by peer”,其实意思很明白了Zookeeper将connection closing.
至于为什么会被关闭connection?那么去检查一下hadoop40机器的zookeeper的日志就会发现一些有用的信息。
四.查看Zookeeper日志:
点击(此处)折叠或打开
2017-10-24 03:25:25,588 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /192.168.100.40 - max is 500
2017-10-24 03:25:25,819 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /192.168.100.40 - max is 500
2017-10-24 03:25:26,092 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x15ed091ee09897d, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:231)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2017-10-24 03:25:26,092 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /192.168.100.40:47084 which had sessionid 0x15ed091ee09897d
2017-10-24 03:25:26,092 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x15ed091ee098981, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:231)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2017-10-24 03:25:26,093 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /192.168.100.40:47304 which had sessionid 0x15ed091ee098981
2017-10-24 03:25:26,093 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
这个时候我们发现"Too many connections from /192.168.100.48 - max is 500",
那就说明这台hadoop40机器的zookeeper的当前connection数已经超过配置的500了,所以新的connection必然被closed,
五.解决方法:
那么我们只需去zookeeper的配置修改一下参数maxClientCnxns调大到1000,重启zookeeper服务生效配置即可。
java.io.IOException: Connection reset by peer相关推荐
- Netty : java.io.IOException: Connection reset by peer
文章目录 1.背景 1.背景 netty聊天室程序,聊天页面如下: 点击上传图片开始报错: 最大上传限制 后来报错: java.io.IOException: Connection reset by ...
- Spark:java.io.IOException: Connection reset by peer
1.背景 # 2.问题2 java.io.IOException: Connection reset by peerat sun.nio.ch.FileDispatcherImpl.read0(Nat ...
- [记录] ---阿里云java.io.IOException: Connection reset by peer的问题
项目部署到阿里云,突然报错,频繁的打印堆栈信息,一开始是把堆内存打满导致服务一直重启,调大堆内存后就不影响正常服务了,但还是一直打堆栈,虽说日志会自动清理,但一直打这个信息着实不好看. 最终在阿里的e ...
- Netty 中 IOException: Connection reset by peer 与 java.nio.channels.ClosedChannelException: null
最近发现系统中出现了很多 IOException: Connection reset by peer 与 ClosedChannelException: null 深入看了看代码, 做了些测试, 发现 ...
- java.net.SocketException: Connection reset by peer: socket write error
最近在做自己的毕业设计,名字是"基于Android的校园二手商品交易系统",于是想把电脑作为服务器,以Android手机作为客户端,做一个C/S结构的程序. 可是一开始就遇到了一个 ...
- IO Error: Connection reset by peer, Authentication lapse 140354 ms.问题的解决
今天在公司服务器上搭建新环境,部署项目时,一切准备就绪了,但是却发现项目总是启动失败,启动日志报:"IO Error: Connection reset by peer, Authentic ...
- Io 异常: Connection reset;Io 异常: Got minus one from a read call;IO 错误: Connection reset by peer, Authe
报错信息: 1. IO 错误: Connection reset by peer, Authentication lapse 0 ms. 搜索网上有些回答说 是JVM随机数造成的延迟问题. 看下ket ...
- JDBC Connection Fails With SEVERE: IO Error: Connection Reset By Peer解决方法
问题描述: 2021.10.20 研发同事反馈一个关于JDBC Connection错误的问题,具体情况如下: JDBC connection with TCPS from Orcle Enterpr ...
- javax.mail.FolderClosedException: * BYE JavaMail Exception: java.io.IOException: Connection dropped
解决方案: (1) 将floder idle设置为true:folder.idle(true); 这样其实只是忽略异常而已: (2) 解决方案二:ReadMail operator need to h ...
- org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
完整错误信息: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipeat org.a ...
最新文章
- spring boot hello world 搭建
- python输出去空格_python输出怎么取消空格
- python打印表情
- 中消协:要选择优质电子显示产品 OLED屏整体上略胜一筹
- 记模拟面试日记(更新...)
- dw选项卡代码_Dreamweaver中js实现竖向选项卡或滑动门代码!急!!!
- Oracle OCP题库变了,052全新题库收集整理-30
- Farrago for Mac 1.1.0 音频效果和音乐剪辑工具 破解版下载
- 数据分析 第七篇:方差分析(单因素方差分析)
- UE4_C++_自定义细节面板_Customizing detail panels
- 三层交换机实现不同vlan间通信
- 金盘系统无法连接服务器,西数金盘Gold系列主要面向企业级服务器及存储系统...
- RedHat下载安装JDK的方法(方法二)
- 【CCTC 2017】来自Mesosphere、美团点评、hyper.sh、凡泰极客、深圳睿云的容器与运维技术实战...
- 微软裁员和.NET的开源
- 小程序tab选项卡请求我的订单数据筛选处理(待收货、代发货、已完成等)
- html网页文件保存rss订阅,博客RSS订阅一些优化方法
- 网易云音乐(一)爬取全部歌手及歌手id
- 2021-06-11认识征信、了解征信,个人征信有什么影响?
- 复杂网络中重要节点挖掘方法综述