Spark2 Failed to send RPC 5346982634 to /ns1:58312: java.nio.channels.ClosedChannelException

将spark任务运行与yarn上出现以下错误：

scala> 18/11/21 16:20:11 ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!
18/11/21 16:20:11 ERROR client.TransportClient: Failed to send RPC 5346982634168622865 to /192.168.88.155:58312: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
18/11/21 16:20:11 ERROR cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful
java.io.IOException: Failed to send RPC 5346982634168622865 to /192.168.88.155:58312: java.nio.channels.ClosedChannelException
at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237)

1.因为spark on yarn，首先查看ResourceMangaer的日志：

2018-11-21 16:20:12,048 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Null container completed...
2018-11-21 16:20:14,714 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Null container completed...

说明container出现了问题，但具体原因尚不知

2.查看NodeMangaer日志：

2018-11-21 16:19:49,777 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=7203,containerID=container_1542787555988_0001_01_000001] is running beyond virtual memory limits. Current usage: 173.8 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.

很明显的看出container使用的虚拟内存超过了设置的2.1G

container使用的虚拟内存是由以下公式计算的：

虚拟内存=yarn.scheduler.minimum-allocation-mb * yarn.nodemanager.vmem-pmem-ratio

参数都是在yarn-site.xml中配置的，如果需要使用的虚拟内存总量超过这个公式计算的值，就会Killing container.

此外，我的yarn.scheduler.minimum-allocation-mb值并没有设置，因此默认为1G，yarn.nodemanager.vmem-pmem-ratio也没设置，默认为2.1，所以就出现了日志中的用了1G里的360M物理内存，用了2.1G里的2.4G虚拟内存。

然后修改yarn-site.xml如下配置

<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>9000</value>
<discription>每个任务最多可用内存,默认8182MB</discription>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>3072</value>
<discription>每个任务最小可用内存</discription>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>3</value> ###物理内存和虚拟内存比率
</property>

Spark2 Failed to send RPC 5346982634 to /ns1:58312: java.nio.channels.ClosedChannelException相关推荐

Spark报错 Failed to send RPC xxx to/ip:43074 java.nio.channels.ClosedChannelException
1.美图 2.背景是这样的,我写了一个spark程序,然后,运行的时候,我kiill yarn掉这个程序,然后发现程序报个错然后继续运行了. 我在页面点击任务停止就没事,我页面点击是调用yarn a ...
【Hadoop】java.io.IOException: Failed on local exception: java.nio.channels.ClosedByInterruptException
查看日志中的报错信息如下: java.io.IOException: Failed on local exception: java.nio.channels.ClosedByInterruptExc ...
SparkStructured报错：Failed to send RPC xxx to /xxx:34744: ClosedChannelException Lost executor
1.视界 2.背景做kafka输出到多个sink,测试类:com.spark.test.sink.multiple.MultipleSinkSumUpdateTest#multipleSinkTes ...
kafka Failed to send producer
线上环境出现的问题版本:kafka-0.10.2.1 现象: 2017-11-29/14:45:02.937/CST WARN [kafka.utils.Logging$class.warn(Log ...
pycharm中报错：Error: failed to send plot to http://127.0.0.1:63342
pycharm中报错:Error: failed to send plot to http://127.0.0.1:63342 import matplotlib.pyplot as plt impo ...
搭建elsticsearch集群报错 failed to send join request to master [{data-node-0}(已经测试)
版权声明:本文为博主原创文章,未经博主允许不得转载. https://blog.csdn.net/qq_24879495/article/details/77718032 搭建elsticsea ...
Nifi出现Failed to send StandardFlowFileRecord xxxx to Hbase due to Failed 3 actions解决
Failed to send StandardFlowFileRecord xxxx to Hbase due to Failed 3 actions: book: 3 times, ;routing ...
kafka Failed to send messages after 3 tries 问题解决
kafka Failed to send messages after 3 tries. 在kafka0.8开发过程中生产者测试用例碰到了 Exception in thread "mai ...
【Flink】Flink 写入 kafka 报错 Failed to send data to Kafka: Expiring 4 record(s) for 20001 ms has passed
文章目录 1.场景1 1.1 概述 1.2 百度 1.3 同样问题 2.场景再现本文为博主九师兄(QQ:541711153 欢迎来探讨技术)原创文章,未经允许博主不允许转载. 1.场景1 1.1 概 ...

Spark2 Failed to send RPC 5346982634 to /ns1:58312: java.nio.channels.ClosedChannelException

Spark2 Failed to send RPC 5346982634 to /ns1:58312: java.nio.channels.ClosedChannelException相关推荐

最新文章

热门文章