spark-submit参数说明--on YARN

示例： spark-submit [--option value] <application jar> [application arguments]

参数名称	含义
--master MASTER_URL	yarn
--deploy-mode DEPLOY_MODE	Driver程序运行的地方：client、cluster
--class CLASS_NAME	The FQCN of the class containing the main method of the application. For example, org.apache.spark.examples.SparkPi. 应用程序主类名称，含包名
--name NAME	应用程序名称
--jars JARS	Driver和Executor依赖的第三方jar包
--properties-file FILE	应用程序属性的文件路径，默认是conf/spark-defaults.conf
以下设置Driver
--driver-cores NUM	Driver程序使用的CPU核数(只用于cluster)，默认为1
--driver-memory MEM	Driver程序使用内存大小
--driver-library-path	Driver程序的库路径
--driver-class-path	Driver程序的类路径
--driver-java-options
以下设置Executor
--num-executors NUM	The total number of YARN containers to allocate for this application. Alternatively, you can use the spark.executor.instances configuration parameter. 启动的executor的数量，默认为2
--executor-cores NUM	Number of processor cores to allocate on each executor 每个executor使用的CPU核数，默认为1
--executor-memory MEM	The maximum heap size to allocate to each executor. Alternatively, you can use the spark.executor.memory configuration parameter. 每个executor内存大小，默认为1G
--queue QUEUE_NAME	The YARN queue to submit to. 提交应用程序给哪个YARN的队列，默认是default队列
--archives ARCHIVES
--files FILES	用逗号隔开的要放置在每个executor工作目录的文件列表

1.部署模式概述

In YARN, each application instance has an ApplicationMaster process, which is the first container started for that application.

The application is responsible for requesting resources from the ResourceManager, and, when allocated them, instructing NodeManagers to start containers on its behalf.

ApplicationMasters obviate the need for an active client — the process starting the application can terminate and coordination continues from a process managed by YARN running on the cluster.

2.部署模式：Cluster

In cluster mode, the driver runs in the ApplicationMaster on a cluster host chosen by YARN.

This means that the same process, which runs in a YARN container, is responsible for both driving the application and requesting resources from YARN.

The client that launches the application doesn't need to continue running for the entire lifetime of the application.

Cluster mode is not well suited to using Spark interactively.

Spark applications that require user input, such as spark-shell and pyspark, need the Spark driver to run inside the client process that initiates the Spark application.

3.部署模式：Client

In client mode, the driver runs on the host where the job is submitted.

The ApplicationMaster is merely present to request executor containers from YARN.

The client communicates with those containers to schedule work after they start:

4.参考文档：

https://www.cloudera.com/documentation/enterprise/5-4-x/topics/cdh_ig_running_spark_on_yarn.html

http://spark.apache.org/docs/1.3.0/running-on-yarn.html

转载于:https://www.cnblogs.com/liugh/p/6953010.html

spark-submit参数说明--on YARN相关推荐

spark：spark-submit 提交任务及参数说明（yarn）
Spark:对于提交命令的理解: https://blog.csdn.net/weixin_38750084/article/details/106973247 spark-submit 可以提交任务 ...
zeppelin报错master is set as yarn, but spark.submit.deployMode is not specified
具体报错如下: java.lang.RuntimeException: master is set as yarn, but spark.submit.deployMode is not specif ...
Spark On Yarn --jars/spark.yarn.jars 踩坑与提升spark submit速度
一.引言 Spark submit 可以选择 --jars 传入本地的jar也可以 --jars 传入HDFS的jar包,经过半下午的实验,终于搞清了两者的关系以及 spark.yarn.jars 和 ...
平台搭建---Spark提交应用程序---Spark Submit提交应用程序及yarn
本部分来源,也可以到spark官网查看英文版. spark-submit 是在spark安装目录中bin目录下的一个shell脚本文件,用于在集群中启动应用程序(如***.py脚本):对于spark支 ...
spark submit参数及调优
spark submit参数及调优原文地址 spark submit参数介绍你可以通过spark-submit --help或者spark-shell --help来查看这些参数. 使用格式: ...
spark submit参数及调优(转载)
spark submit参数介绍你可以通过spark-submit --help或者spark-shell --help来查看这些参数. 使用格式: ./bin/spark-submit \--c ...
spark submit参数及调试
原文:http://www.cnblogs.com/haoyy/p/6893943.html spark submit参数介绍你可以通过spark-submit --help或者spark-shel ...
Spark Submit提交及参数调优
https://www.cnblogs.com/LHWorldBlog/p/8414342.html https://www.cnblogs.com/haoyy/p/6893943.html spar ...
Spark：spark submit参数及调优 perfect
先看下spark运行原理流程: 我们使用spark-submit提交一个Spark作业之后,这个作业就会启动一个对应的Driver进程. 根据你使用的部署模式(deploy-mode)不同,Drive ...
【原创】大数据基础之Spark（1）Spark Submit即Spark任务提交过程
Spark2.1.1 一 Spark Submit本地解析 1.1 现象提交命令: spark-submit --master local[10] --driver-memory 30g --cla ...

spark-submit参数说明--on YARN

spark-submit参数说明--on YARN相关推荐

最新文章

热门文章