spark.yarn.archive spark.yarn.jars

参考自：http://spark.apache.org/docs/latest/running-on-yarn.html#preparations

在spark on yarn模式下，/usr/local/spark-current2.3/conf下的spark-defaults.conf配置文件有一个spark.yarn.archive配置项

1.如果没有配置

#spark.yarn.archive hdfs://ddd/project/spark2.3_test_liuzc/jar

程序运行时日志：

20/03/25 16:18:10 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
20/03/25 16:18:16 INFO yarn.Client: Uploading resource file:/tmp/spark-8b8a4eb9-365e-403b-9411-abc60c3ea461/__spark_libs__5628865856189033955.zip -> hdfs://ddd/user/pgxl/.sparkStaging/application_1570676918196_86932/__spark_libs__5628865856189033955.zip

2.如果已经配置

spark.yarn.archive hdfs://ddd/project/spark2.3_test_liuzc/jar

运行时日志：

20/03/25 16:21:36 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://ddd/project/spark2.3_test_liuzc/jar

总结：(个人理解)

executor端运行所需的jar包,如果没有配置，会从本地把jar包压缩上传到一个hdfs临时目录，

如果已经配置的话，会直接从hdfs路径上读取，可以节省一点时间

spark.yarn.archive spark.yarn.jars相关推荐

Neither spark.yarn.jars nor spark.yarn.archive is set
完整报错如下: (Python3.6) appleyuchi@Desktop:target$ spark-submit --class WordCountLocal --master yarn --d ...
添加spark.yarn.jars 解决 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set,
一:问题现象: 在spark on yarn 提交任务是,提示如下: WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive ...
yarn集群下启动spark错误WARN:66 - Neither spark.yarn.jars nor spark.yarn.archive is set
yarn集群下启动spark错误如下: WARN Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling ...
spark.yarn.archive 的正确设置方法
spark.yarn.archive 的设置方法有很多,下面三种 ①spark.yarn.jars hdfs://Desktop:9000/spark/jars/* ②spark.yarn.jars ...
Spark通过YARN提交任务不成功（包含YARN cluster和YARN client)
无论用YARN cluster和YARN client来跑,均会出现如下问题. [spark@master spark-1.6.1-bin-hadoop2.6]$ jps 2049 NameNode ...
java spark yarn_《Spark官方文档》在YARN上运行Spark
原文链接 Spark在 0.6.0版本后支持在YARN(hadoop NextGen)上运行,并且在后续版本中不断改进. 在YARN上启动Spark 首先,确认 HADOOP_CONF_DIR或YAR ...
spark提交到yarn_详细总结spark基于standalone、yarn集群提交作业流程
最近总结了一些关于spark core的内容,今天先来和大家分享一下spark的运行模式. spark运行模式 (1)local:在本地eclipse.IDEA中写spark代码运行程序,一般用于测试 ...
Spark 运行模式 standalong yarn
standalong 模式需要在spark master 节点上启动 spark/sbin/start-all.sh 主从节点都可以run standalong client ./bin/spark- ...
zeppelin报错master is set as yarn, but spark.submit.deployMode is not specified
具体报错如下: java.lang.RuntimeException: master is set as yarn, but spark.submit.deployMode is not specif ...

spark.yarn.archive spark.yarn.jars

spark.yarn.archive spark.yarn.jars相关推荐

最新文章

热门文章