Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterva

问题描述：提交spark-submit时，在ssh终端提交没有问题，但是在代码中ssh登陆后，再提交命令就出现以下问题了，开始怀疑是用户问题，但是如果是用户问题，那么我在ssh终端同一个用户执行又执行正确，故排除了此情况。第二感觉是环境变量，我在.sh添加环境后，还是报错。

sh文件

#!/bin/bash
#调用java 程序需要添加上，否则直接跳出shell
#export JAVA_HOME=/usr/java/jdk1.8.0_201
export JAVA_HOME=/usr/java/jdk1.8
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/binarg1=$1
arg2=$2
arg3=$3
arg4=$4spark-submit \--master yarn \--deploy-mode client \--class Main  \--executor-cores 6 \--executor-memory 10G \--num-executors 8 \--conf spark.driver.cores=6 \--conf spark.driver.maxResultSize=0 \--conf spark.driver.memony=5g \--jars hdfs://bdpcluster/business/oozie/lib/*.jar,hdfs://bdpcluster/business/oozie/hub/lib/* \hdfs://bdpcluster/business/oozie/dhe-dipc.jar \"$arg1" "$arg2" "$arg3" "$arg4"

错误信息

org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatIntervalat org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:47)at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:62)at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:58)at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76)at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:92)at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:785)at org.apache.spark.executor.Executor$$anon$2$$anonfun$run$1.apply$mcV$sp(Executor.scala:814)at org.apache.spark.executor.Executor$$anon$2$$anonfun$run$1.apply(Executor.scala:814)at org.apache.spark.executor.Executor$$anon$2$$anonfun$run$1.apply(Executor.scala:814)at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1992)at org.apache.spark.executor.Executor$$anon$2.run(Executor.scala:814)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10 seconds]at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:201)at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)... 14 more19/05/31 11:17:01 ERROR ApplicationMaster: Uncaught exception:
org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120 seconds]. This timeout is controlled by spark.rpc.askTimeout
at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:47)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:62)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:58)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76)
at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:92)
at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:76)
at org.apache.spark.deploy.yarn.YarnAllocator.<init>(YarnAllocator.scala:103)
at org.apache.spark.deploy.yarn.YarnRMClient.register(YarnRMClient.scala:78)
at org.apache.spark.deploy.yarn.ApplicationMaster.registerAM(ApplicationMaster.scala:462)
at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:534)
at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:347)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$5.run(ApplicationMaster.scala:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:814)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:259)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:839)
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:869)
at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [120 seconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:201)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
... 19 more

ssh终端提交：

/home/xx.sh “2019-05-01 00:00:00” “2019-05-31 00:00:00” “week”
“28,29,30,33,43,74,77”

执行正常，spark没有报错的的task

java代码提交关键代码：
SshUtil rec = new SshUtil(DATAX_HOST_LIST.get(0), DATAX_USER, DATAX_PASSWORD, DATAX_PORT);

        String script = "source /etc/profile;/home/x.sh \"2019-05-01 00:00:00\" \"2019-06-01 00:00:00\" \"month\" \"28,29,30,33,43,74,77\"";

解决方案：
java代码提交命令更改为nohub，即

     script = "source /etc/profile;nohup /home/x.sh \"2019-05-01 00:00:00\" \"2019-06-01 00:00:00\" \"month\" \"28,29,30,33,43,74,77\" &";打印标准和输出日志：script = "source /etc/profile;nohup /home/x.sh \"2019-05-01 00:00:00\" \"2019-06-01 00:00:00\" \"month\" \"28,29,30,33,43,74,77\" >>/home/dev/debug.log 2>&1 &";

Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterva相关推荐

从Daemons到finalize timed out after 10 seconds
〇.序起因,在Fabric遇到了很多BUG..出于想对于BUG的来龙去脉做进一步的了解.做了一番调查.本文也是Android Crash的防护与追踪的后续研究. 这是Fabric给出的一些link ...
OPPO 设备报错 android.content.res.AssetManager.finalize() timed out after 120 seconds
这是从后台看到的一个错误日志,在一些OPPO 机型会报错 ,如 R9 等. 可以看到出错的设备基本是OPPO 的一些设备,推测应该是极光的SDK在部分OPPO设备导致,在极光论坛搜索该错误能找到很多帖 ...
Nagios的警报信息CHECK_NRPE: Socket timeout after 10 seconds.
转载自:http://blog.chinaunix.net/uid-26446098-id-3292730.html Notification Type: PROBLEM Service: ora_l ...
异常-java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
异常-java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds] 参考文章: (1)异常- ...
Kettle 6.1：Triggering heartbeat signal for ... at every 10 seconds 问题解决
在数据迁移过程中执行yi执行一段时间发警告 trggering heartbeat signal for ... at every 10 seconds 并无法继续执行. 百思不得其解,数据源 sql ...
腾讯云函数报错 Invoking task timed out after 3 seconds
具体报错: {"errorCode":-1,"errorMessage":"Invoking task timed out after 3 secon ...
Docker pull unexpected EOF Retrying in 10 seconds
手贱docker rmi ,docker rm掉了服务器上的镜像和容器再docker pull就发生了标题的报错 Retrying in 10 seconds unexpected EOF [roo ...
Spark Executor heartbeat timed out
Spark Executor heartbeat timed out after问题排查. 百度调整参数:如下图参数没有用. 解决办法: 代码排查问题顺序调整,解决问题.
关于Flash Player 10 socket connection timeout
转载请注明,来自:http://blog.csdn.net/skyman_2001 今天我在调试游戏的时候,发现一个有趣的现象,就是flash连接上服务器后,服务器关闭连接,flash会触发secur ...

Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterva

Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterva相关推荐

最新文章

热门文章