Submitting Applications

The spark-submit script in Spark’s bin directory is used to launch applications on a cluster. It can use all of Spark’s supported cluster managersthrough a uniform interface so you don’t have to configure your application specially for each one.

Bundling Your Application’s Dependencies

If your code depends on other projects, you will need to package them alongside your application in order to distribute the code to a Spark cluster. To do this, create an assembly jar (or “uber” jar) containing your code and its dependencies. Both sbt and Maven have assembly plugins. When creating assembly jars, list Spark and Hadoop as provided dependencies; these need not be bundled since they are provided by the cluster manager at runtime. Once you have an assembled jar you can call the bin/spark-submit script as shown here while passing your jar.

For Python, you can use the --py-files argument of spark-submit to add .py.zip or .egg files to be distributed with your application. If you depend on multiple Python files we recommend packaging them into a .zip or .egg.

Launching Applications with spark-submit

Once a user application is bundled, it can be launched using the bin/spark-submit script. This script takes care of setting up the classpath with Spark and its dependencies, and can support different cluster managers and deploy modes that Spark supports:

./bin/spark-submit \--class <main-class> \--master <master-url> \--deploy-mode <deploy-mode> \--conf <key>=<value> \... # other options<application-jar> \[application-arguments]

Some of the commonly used options are:

  • --class: The entry point for your application (e.g. org.apache.spark.examples.SparkPi)
  • --master: The master URL for the cluster (e.g. spark://23.195.26.187:7077)
  • --deploy-mode: Whether to deploy your driver on the worker nodes (cluster) or locally as an external client (client) (default: client
  • --conf: Arbitrary Spark configuration property in key=value format. For values that contain spaces wrap “key=value” in quotes (as shown).
  • application-jar: Path to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an hdfs:// path or a file:// path that is present on all nodes.
  • application-arguments: Arguments passed to the main method of your main class, if any

 A common deployment strategy is to submit your application from a gateway machine that is physically co-located with your worker machines (e.g. Master node in a standalone EC2 cluster). In this setup, client mode is appropriate. In client mode, the driver is launched directly within the spark-submit process which acts as a client to the cluster. The input and output of the application is attached to the console. Thus, this mode is especially suitable for applications that involve the REPL (e.g. Spark shell).

Alternatively, if your application is submitted from a machine far from the worker machines (e.g. locally on your laptop), it is common to usecluster mode to minimize network latency between the drivers and the executors. Currently, standalone mode does not support cluster mode for Python applications.

For Python applications, simply pass a .py file in the place of <application-jar> instead of a JAR, and add Python .zip.egg or .py files to the search path with --py-files.

There are a few options available that are specific to the cluster manager that is being used. For example, with a Spark standalone cluster withcluster deploy mode, you can also specify --supervise to make sure that the driver is automatically restarted if it fails with non-zero exit code. To enumerate all such options available to spark-submit, run it with --help. Here are a few examples of common options:

# Run application locally on 8 cores
./bin/spark-submit \--class org.apache.spark.examples.SparkPi \--master local[8] \/path/to/examples.jar \100# Run on a Spark standalone cluster in client deploy mode
./bin/spark-submit \--class org.apache.spark.examples.SparkPi \--master spark://207.184.161.138:7077 \--executor-memory 20G \--total-executor-cores 100 \/path/to/examples.jar \1000# Run on a Spark standalone cluster in cluster deploy mode with supervise
./bin/spark-submit \--class org.apache.spark.examples.SparkPi \--master spark://207.184.161.138:7077 \--deploy-mode cluster \--supervise \--executor-memory 20G \--total-executor-cores 100 \/path/to/examples.jar \1000# Run on a YARN cluster
export HADOOP_CONF_DIR=XXX
./bin/spark-submit \--class org.apache.spark.examples.SparkPi \--master yarn \--deploy-mode cluster \  # can be client for client mode--executor-memory 20G \--num-executors 50 \/path/to/examples.jar \1000# Run a Python application on a Spark standalone cluster
./bin/spark-submit \--master spark://207.184.161.138:7077 \examples/src/main/python/pi.py \1000# Run on a Mesos cluster in cluster deploy mode with supervise
./bin/spark-submit \--class org.apache.spark.examples.SparkPi \--master mesos://207.184.161.138:7077 \--deploy-mode cluster \--supervise \--executor-memory 20G \--total-executor-cores 100 \http://path/to/examples.jar \1000

Master URLs

The master URL passed to Spark can be in one of the following formats:

Master URL Meaning
local Run Spark locally with one worker thread (i.e. no parallelism at all).
local[K] Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine).
local[*] Run Spark locally with as many worker threads as logical cores on your machine.
spark://HOST:PORT Connect to the given Spark standalone cluster master. The port must be whichever one your master is configured to use, which is 7077 by default.
mesos://HOST:PORT Connect to the given Mesos cluster. The port must be whichever one your is configured to use, which is 5050 by default. Or, for a Mesos cluster using ZooKeeper, use mesos://zk://.... To submit with --deploy-mode cluster, the HOST:PORT should be configured to connect to the MesosClusterDispatcher.
yarn Connect to a YARN cluster in client or cluster mode depending on the value of --deploy-mode. The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable.

Loading Configuration from a File

The spark-submit script can load default Spark configuration values from a properties file and pass them on to your application. By default it will read options from conf/spark-defaults.conf in the Spark directory. For more detail, see the section on loading default configurations.

Loading default Spark configurations this way can obviate the need for certain flags to spark-submit. For instance, if the spark.master property is set, you can safely omit the --master flag from spark-submit. In general, configuration values explicitly set on a SparkConf take the highest precedence, then flags passed to spark-submit, then values in the defaults file.

If you are ever unclear where configuration options are coming from, you can print out fine-grained debugging information by running spark-submit with the --verbose option.

Advanced Dependency Management

When using spark-submit, the application jar along with any jars included with the --jars option will be automatically transferred to the cluster. URLs supplied after --jars must be separated by commas. That list is included on the driver and executor classpaths. Directory expansion does not work with --jars.

Spark uses the following URL scheme to allow different strategies for disseminating jars:

  • file: - Absolute paths and file:/ URIs are served by the driver’s HTTP file server, and every executor pulls the file from the driver HTTP server.
  • hdfs:http:https:ftp: - these pull down files and JARs from the URI as expected
  • local: - a URI starting with local:/ is expected to exist as a local file on each worker node. This means that no network IO will be incurred, and works well for large files/JARs that are pushed to each worker, or shared via NFS, GlusterFS, etc.

Note that JARs and files are copied to the working directory for each SparkContext on the executor nodes. This can use up a significant amount of space over time and will need to be cleaned up. With YARN, cleanup is handled automatically, and with Spark standalone, automatic cleanup can be configured with the spark.worker.cleanup.appDataTtl property.

Users may also include any other dependencies by supplying a comma-delimited list of maven coordinates with --packages. All transitive dependencies will be handled when using this command. Additional repositories (or resolvers in SBT) can be added in a comma-delimited fashion with the flag --repositories. These commands can be used with pysparkspark-shell, and spark-submit to include Spark Packages.

For Python, the equivalent --py-files option can be used to distribute .egg.zip and .py libraries to executors.

More Information

Once you have deployed your application, the cluster mode overview describes the components involved in distributed execution, and how to monitor and debug applications.

Submitting Applications相关推荐

  1. Spark 1.1.1 Submitting Applications

    回到目录 Submitting Applications The spark-submit script in Spark's bin directory is used to launch appl ...

  2. spark学习(二)

    Spark是一个通用的并行计算框架,由UCBerkeley的AMP实验室开发. Spark和Hadoop有什么不同呢? Spark是基于map reduce算法实现的分布式计算,拥有Hadoop Ma ...

  3. spark的三种运行模式以及yarn-client和yarn-cluster在提交命令上的区别

    本文针对的是Spark 2.3.1 standalone:线下模式 分为standalone-client和standalone-cluster两种模式 yarn:线上模式 又分为yarn-clien ...

  4. Spark集群的启动日志

    Created by Wang, Jerry, last modified on Aug 24, 2015 added by Jerry:- /root/devExpert/spark-1.4.1/s ...

  5. Spark2.1.0 + CarbonData1.0.0集群模式部署及使用入门

    1 引言 Apache CarbonData是一个面向大数据平台的基于索引的列式数据格式,由华为大数据团队贡献给Apache社区,目前最新版本是1.0.0版.介于目前主流大数据组件应用场景的局限性,C ...

  6. spark执行优化——依赖上传到HDFS二(-conf spark.yarn.dist.jars或者--jars 的使用)

    1.说明 之前整理过一篇类似文章,但是这个spark.yarn.jar配置的目录最好只是放spark jars目录下的jar包,如果放入其他的jar包,很大概率会有冲突,而且如果项目比较多,jar包引 ...

  7. 怎么编写提醒人结账的短信息_我如何在短短8个月的时间里从编写第一行代码到获得$ 226K的工作机会...

    怎么编写提醒人结账的短信息 I got offers from Google, Lyft, Yelp, cloud unicorn Rubrik, IBM Artificial Intelligenc ...

  8. docker-compose up:ERROR: Encountered errors while bringing up the project.錯誤及解決方式

    docker-compose up:ERROR: Encountered errors while bringing up the project.錯誤及解決方式 前言 發生原因 錯誤訊息 解決方式 ...

  9. Spark 学习【二】

    Spark Core [05-07] 两个demo 很多复杂的业务拆分开都是变种的wc: 分组 ==> 变种WC ==> 数据补齐 将不同组的数据按规则合并在一起 split(" ...

  10. Spark 2.4.8 提交应用

    Spark 2.4.8 提交应用 Submitting Applications 捆绑你应用程序的依赖 用spark-submit启动应用程序 Master URLs 从文件中下载配置文件 高级依赖管 ...

最新文章

  1. cookie用法之一,最简单cookie操作
  2. matlab错误103 577,小编为你作答win7系统安装MATLAB出现 license manager error 103错误的还原教程...
  3. 爬取网页时自动获取网页编码信息,并对特殊的乱码页面(压缩过的网页内容)用gzip进行解码。...
  4. 【Linux】Linux服务器(centos7)环境搭建java/python3/nginx
  5. html中图片的属性优化,Html标签元素在SEO中的优化方式(二)
  6. 硬件基础 —— 二极管
  7. 莫比乌斯反演部分题目总结
  8. 交换排序之冒泡排序(java实现)
  9. 记某站被搜索引擎入侵
  10. Tabular Editor学习笔记_1
  11. EJB开发web service
  12. 使用python的turtle绘画滑稽脸
  13. 分销系统如何加粉,三级分销政策,B2B2C分销规则
  14. 5.3.3—二叉查找树—Validate Binary Sear Tree
  15. H263H264MPEG4
  16. 2019-04-07 Python之利用PIL改变图片颜色和生成手绘图
  17. 一、TF2 常用命令
  18. java后台图片大小压缩
  19. 天馈线测试仪 是什么 具备什么样的功能
  20. C#的System.Diagnostics.Trace.WriteLine 写入文件

热门文章

  1. Volume Manager for mac 详细教程
  2. TeamViewer 远程设备的画面黑屏怎么办?
  3. JS 函数中arguments的使用
  4. 如何用Camtasia进行内容补充?
  5. 创建Maven项目时提示web.xml is missing and failOnMissingWebXml is set to true错误解决方案
  6. Serverless Computing:现状与基础知识
  7. React 事件处理函数
  8. (转)oop设计模式简介
  9. [蛋蛋四格漫画]-贺沪江日语四周年版庆
  10. 苹果mac窗口速调辅助工具:Magnet