1、下载Spark安装包

官网网址：http://spark.apache.org/downloads.html

2、Spark安装过程

2.1、上传并解压缩

[potter@potter2 ~]$ tar -zxvf spark-2.3.0-bin-hadoop2.7.tgz -C apps/

2.2、修改配置文件

（1）进入配置文件所在目录

/home/potter/apps/spark-2.3.0-bin-hadoop2.7/conf

  1. [potter@potter2 conf]$ ll
  2. total 36
  3. -rw-r–r-- 1 potter potter 996 Feb 23 03:42 docker.properties.template
  4. -rw-r–r-- 1 potter potter 1105 Feb 23 03:42 fairscheduler.xml.template
  5. -rw-r–r-- 1 potter potter 2025 Feb 23 03:42 log4j.properties.template
  6. -rw-r–r-- 1 potter potter 7801 Feb 23 03:42 metrics.properties.template
  7. -rw-r–r-- 1 potter potter 865 Feb 23 03:42 slaves.template
  8. -rw-r–r-- 1 potter potter 1292 Feb 23 03:42 spark-defaults.conf.template
  9. -rwxr-xr-x 1 potter potter 4221 Feb 23 03:42 spark-env.sh.template
（2）修改spark-env.sh文件

复制spark-env.sh.template，并重命名为spark-env.sh，并在文件最后添加配置内容

  1. [potter@potter2 conf]$ cp spark-env.sh.template spark-env.sh
  2. [potter@potter2 conf]$ vi spark-env.sh
  1. export JAVA_HOME=/usr/local/java/jdk1.8.0_73
  2. #export SCALA_HOME=/usr/share/scala
  3. export HADOOP_HOME=/home/potter/apps/hadoop-2.7.5
  4. export HADOOP_CONF_DIR=/home/potter/apps/hadoop-2.7.5/etc/hadoop
  5. export SPARK_WORKER_MEMORY=500m
  6. export SPARK_WORKER_CORES=1
  7. export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=potter2:2181,potter3:2181,potter4:2181,potter5:2181 -Dspark.deploy.zookeeper.dir=/spark"
注：

#export SPARK_MASTER_IP=hadoop1  这个配置要注释掉。

集群搭建时配置的spark参数可能和现在的不一样，主要是考虑个人电脑配置问题，如果memory配置太大，机器运行很慢。

说明：

-Dspark.deploy.recoveryMode=ZOOKEEPER    #说明整个集群状态是通过zookeeper来维护的，整个集群状态的恢复也是通过zookeeper来维护的。就是说用zookeeper做了spark的HA配置，Master(Active)挂掉的话，Master(standby)要想变成Master（Active）的话，Master(Standby)就要像zookeeper读取整个集群状态信息，然后进行恢复所有Worker和Driver的状态信息，和所有的Application状态信息；

-Dspark.deploy.zookeeper.url=potter2:2181,potter3:2181,potter4:2181,potter5:2181#将所有配置了zookeeper，并且在这台机器上有可能做master(Active)的机器都配置进来；（我用了4台，就配置了4台）

-Dspark.deploy.zookeeper.dir=/spark

这里的dir和zookeeper配置文件zoo.cfg中的dataDir的区别？？？

-Dspark.deploy.zookeeper.dir是保存spark的元数据，保存了spark的作业运行状态；

zookeeper会保存spark集群的所有的状态信息，包括所有的Workers信息，所有的Applactions信息，所有的Driver信息,如果集群
（3）复制slaves.template变成slaves
  1. [potter@potter2 conf]$ cp slaves.template slaves
  2. [potter@potter2 conf]$ vi slaves

添加以下内容：

  1. potter2
  2. potter3
  3. potter4
  4. potter5
（4）将安装包分发给其他节点
  1. [potter@potter2 apps]$ scp -r spark-2.3.0-bin-hadoop2.7/ potter3:PWD&lt;/div&gt;&lt;/div&gt;&lt;/li&gt;&lt;li&gt;&lt;divclass=&quot;hljs−ln−numbers&quot;&gt;&lt;divclass=&quot;hljs−ln−linehljs−ln−n&quot;data−line−number=&quot;2&quot;&gt;&lt;/div&gt;&lt;/div&gt;&lt;divclass=&quot;hljs−ln−code&quot;&gt;&lt;divclass=&quot;hljs−ln−line&quot;&gt;[potter&lt;spanclass=&quot;hljs−meta&quot;&gt;@potter&lt;/span&gt;2apps]PWD&lt;/div&gt;&lt;/div&gt;&lt;/li&gt;&lt;li&gt;&lt;div class=&quot;hljs-ln-numbers&quot;&gt;&lt;div class=&quot;hljs-ln-line hljs-ln-n&quot; data-line-number=&quot;2&quot;&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;hljs-ln-code&quot;&gt;&lt;div class=&quot;hljs-ln-line&quot;&gt;[potter&lt;span class=&quot;hljs-meta&quot;&gt;@potter&lt;/span&gt;2 apps]PWD</div></div></li><li><divclass="hljs−ln−numbers"><divclass="hljs−ln−linehljs−ln−n"data−line−number="2"></div></div><divclass="hljs−ln−code"><divclass="hljs−ln−line">[potter<spanclass="hljs−meta">@potter</span>2apps] scp -r spark-2.3.0-bin-hadoop2.7/ potter4:PWD&lt;/div&gt;&lt;/div&gt;&lt;/li&gt;&lt;li&gt;&lt;divclass=&quot;hljs−ln−numbers&quot;&gt;&lt;divclass=&quot;hljs−ln−linehljs−ln−n&quot;data−line−number=&quot;3&quot;&gt;&lt;/div&gt;&lt;/div&gt;&lt;divclass=&quot;hljs−ln−code&quot;&gt;&lt;divclass=&quot;hljs−ln−line&quot;&gt;[potter&lt;spanclass=&quot;hljs−meta&quot;&gt;@potter&lt;/span&gt;2apps]PWD&lt;/div&gt;&lt;/div&gt;&lt;/li&gt;&lt;li&gt;&lt;div class=&quot;hljs-ln-numbers&quot;&gt;&lt;div class=&quot;hljs-ln-line hljs-ln-n&quot; data-line-number=&quot;3&quot;&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;hljs-ln-code&quot;&gt;&lt;div class=&quot;hljs-ln-line&quot;&gt;[potter&lt;span class=&quot;hljs-meta&quot;&gt;@potter&lt;/span&gt;2 apps]PWD</div></div></li><li><divclass="hljs−ln−numbers"><divclass="hljs−ln−linehljs−ln−n"data−line−number="3"></div></div><divclass="hljs−ln−code"><divclass="hljs−ln−line">[potter<spanclass="hljs−meta">@potter</span>2apps] scp -r spark-2.3.0-bin-hadoop2.7/ potter5:PWD&lt;/div&gt;&lt;/div&gt;&lt;/li&gt;&lt;/ol&gt;&lt;/code&gt;&lt;divclass=&quot;hljs−button2&quot;data−title=&quot;复制&quot;&gt;&lt;/div&gt;&lt;/pre&gt;&lt;h2&gt;&lt;aname=&quot;t4&quot;&gt;&lt;/a&gt;2.3、配置环境变量&lt;/h2&gt;&lt;preonclick=&quot;hljs.copyCode(event)&quot;&gt;&lt;codeclass=&quot;language−javahljs&quot;&gt;[potter&lt;spanclass=&quot;hljs−meta&quot;&gt;@potter&lt;/span&gt;2]PWD&lt;/div&gt;&lt;/div&gt;&lt;/li&gt;&lt;/ol&gt;&lt;/code&gt;&lt;div class=&quot;hljs-button {2}&quot; data-title=&quot;复制&quot;&gt;&lt;/div&gt;&lt;/pre&gt;&lt;h2&gt;&lt;a name=&quot;t4&quot;&gt;&lt;/a&gt;2.3、配置环境变量&lt;/h2&gt;&lt;pre &quot;hljs.copyCode(event)&quot;&gt;&lt;code class=&quot;language-java hljs&quot;&gt;[potter&lt;span class=&quot;hljs-meta&quot;&gt;@potter&lt;/span&gt;2 ~]PWD</div></div></li></ol></code><divclass="hljs−button2"data−title="复制"></div></pre><h2><aname="t4"></a>2.3、配置环境变量</h2><preonclick="hljs.copyCode(event)"><codeclass="language−javahljs">[potter<spanclass="hljs−meta">@potter</span>2 ] vi .bashrc
  1. export SPARK_HOME=/home/potter/apps/spark-2.3.0-bin-hadoop2.7
  2. export PATH=PATH:PATH:PATH:SPARK_HOME/bin

保存并使其立即生效

[potter@potter2 ~]$ source .bashrc

2.4、配置spark-defaults.conf

Spark默认是本地模式：

进入/home/potter/apps/spark-2.3.0-bin-hadoop2.7/conf

复制一个spark-defaults.conf文件

[potter@potter2 conf]$ cp spark-defaults.conf.template spark-defaults.conf
[potter@potter2 conf]$ vi spark-defaults.conf
  1. # This is useful for setting default environmental settings.
  2. # Example:
  3. spark.master spark://potter2:7077,potter3:7077,potter4:7077,potter5:7077
  4. # spark.eventLog.enabled true
  5. # spark.eventLog.dir hdfs://namenode:8021/directory
  6. # spark.serializer org.apache.spark.serializer.KryoSerializer
  7. # spark.driver.memory 5g
  8. # spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers=“one two three"

分发到其他节点：
  1. [potter@potter2 conf]$ scp -r spark-defaults.conf potter3:PWD&lt;/div&gt;&lt;/div&gt;&lt;/li&gt;&lt;li&gt;&lt;divclass=&quot;hljs−ln−numbers&quot;&gt;&lt;divclass=&quot;hljs−ln−linehljs−ln−n&quot;data−line−number=&quot;2&quot;&gt;&lt;/div&gt;&lt;/div&gt;&lt;divclass=&quot;hljs−ln−code&quot;&gt;&lt;divclass=&quot;hljs−ln−line&quot;&gt;[potter&lt;spanclass=&quot;hljs−meta&quot;&gt;@potter&lt;/span&gt;2conf]PWD&lt;/div&gt;&lt;/div&gt;&lt;/li&gt;&lt;li&gt;&lt;div class=&quot;hljs-ln-numbers&quot;&gt;&lt;div class=&quot;hljs-ln-line hljs-ln-n&quot; data-line-number=&quot;2&quot;&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;hljs-ln-code&quot;&gt;&lt;div class=&quot;hljs-ln-line&quot;&gt;[potter&lt;span class=&quot;hljs-meta&quot;&gt;@potter&lt;/span&gt;2 conf]PWD</div></div></li><li><divclass="hljs−ln−numbers"><divclass="hljs−ln−linehljs−ln−n"data−line−number="2"></div></div><divclass="hljs−ln−code"><divclass="hljs−ln−line">[potter<spanclass="hljs−meta">@potter</span>2conf] scp -r spark-defaults.conf potter4:PWD&lt;/div&gt;&lt;/div&gt;&lt;/li&gt;&lt;li&gt;&lt;divclass=&quot;hljs−ln−numbers&quot;&gt;&lt;divclass=&quot;hljs−ln−linehljs−ln−n&quot;data−line−number=&quot;3&quot;&gt;&lt;/div&gt;&lt;/div&gt;&lt;divclass=&quot;hljs−ln−code&quot;&gt;&lt;divclass=&quot;hljs−ln−line&quot;&gt;[potter&lt;spanclass=&quot;hljs−meta&quot;&gt;@potter&lt;/span&gt;2conf]PWD&lt;/div&gt;&lt;/div&gt;&lt;/li&gt;&lt;li&gt;&lt;div class=&quot;hljs-ln-numbers&quot;&gt;&lt;div class=&quot;hljs-ln-line hljs-ln-n&quot; data-line-number=&quot;3&quot;&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;hljs-ln-code&quot;&gt;&lt;div class=&quot;hljs-ln-line&quot;&gt;[potter&lt;span class=&quot;hljs-meta&quot;&gt;@potter&lt;/span&gt;2 conf]PWD</div></div></li><li><divclass="hljs−ln−numbers"><divclass="hljs−ln−linehljs−ln−n"data−line−number="3"></div></div><divclass="hljs−ln−code"><divclass="hljs−ln−line">[potter<spanclass="hljs−meta">@potter</span>2conf] scp -r spark-defaults.conf potter5:PWD&lt;/div&gt;&lt;/div&gt;&lt;/li&gt;&lt;/ol&gt;&lt;/code&gt;&lt;divclass=&quot;hljs−button2&quot;data−title=&quot;复制&quot;&gt;&lt;/div&gt;&lt;/pre&gt;&lt;h1&gt;&lt;aname=&quot;t6&quot;&gt;&lt;/a&gt;3、启动&lt;/h1&gt;&lt;h2&gt;&lt;aname=&quot;t7&quot;&gt;&lt;/a&gt;3.1、先启动zookeeper集群&lt;/h2&gt;&lt;p&gt;所有节点均要执行&lt;/p&gt;&lt;preonclick=&quot;hljs.copyCode(event)&quot;&gt;&lt;codeclass=&quot;language−javahljs&quot;&gt;&lt;olclass=&quot;hljs−ln&quot;&gt;&lt;li&gt;&lt;divclass=&quot;hljs−ln−numbers&quot;&gt;&lt;divclass=&quot;hljs−ln−linehljs−ln−n&quot;data−line−number=&quot;1&quot;&gt;&lt;/div&gt;&lt;/div&gt;&lt;divclass=&quot;hljs−ln−code&quot;&gt;&lt;divclass=&quot;hljs−ln−line&quot;&gt;[potter&lt;spanclass=&quot;hljs−meta&quot;&gt;@potter&lt;/span&gt;2]PWD&lt;/div&gt;&lt;/div&gt;&lt;/li&gt;&lt;/ol&gt;&lt;/code&gt;&lt;div class=&quot;hljs-button {2}&quot; data-title=&quot;复制&quot;&gt;&lt;/div&gt;&lt;/pre&gt;&lt;h1&gt;&lt;a name=&quot;t6&quot;&gt;&lt;/a&gt;3、启动&lt;/h1&gt;&lt;h2&gt;&lt;a name=&quot;t7&quot;&gt;&lt;/a&gt;3.1、先启动zookeeper集群&lt;/h2&gt;&lt;p&gt;所有节点均要执行&lt;/p&gt;&lt;pre &quot;hljs.copyCode(event)&quot;&gt;&lt;code class=&quot;language-java hljs&quot;&gt;&lt;ol class=&quot;hljs-ln&quot;&gt;&lt;li&gt;&lt;div class=&quot;hljs-ln-numbers&quot;&gt;&lt;div class=&quot;hljs-ln-line hljs-ln-n&quot; data-line-number=&quot;1&quot;&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class=&quot;hljs-ln-code&quot;&gt;&lt;div class=&quot;hljs-ln-line&quot;&gt;[potter&lt;span class=&quot;hljs-meta&quot;&gt;@potter&lt;/span&gt;2 ~]PWD</div></div></li></ol></code><divclass="hljs−button2"data−title="复制"></div></pre><h1><aname="t6"></a>3、启动</h1><h2><aname="t7"></a>3.1、先启动zookeeper集群</h2><p>所有节点均要执行</p><preonclick="hljs.copyCode(event)"><codeclass="language−javahljs"><olclass="hljs−ln"><li><divclass="hljs−ln−numbers"><divclass="hljs−ln−linehljs−ln−n"data−line−number="1"></div></div><divclass="hljs−ln−code"><divclass="hljs−ln−line">[potter<spanclass="hljs−meta">@potter</span>2 ] zkServer.sh start
  2. ZooKeeper JMX enabled by default
  3. Using config: /home/potter/apps/zookeeper-3.4.10/bin/…/conf/zoo.cfg
  4. Starting zookeeper … already running as process 3703.
  5. [potter@potter2 ~]$ zkServer.sh status
  6. ZooKeeper JMX enabled by default
  7. Using config: /home/potter/apps/zookeeper-3.4.10/bin/…/conf/zoo.cfg
  8. Mode: follower

3.2、启动HDFS集群

任意一个节点执行即可

[potter@potter2 ~]$ start-dfs.sh

3.3、再启动Spark集群

  1. [potter@potter2 ~]$ cd apps/spark-2.3.0-bin-hadoop2.7/sbin/
  2. [potter@potter2 sbin]$ ./start-all.sh

3.4、查看进程

  1. [potter@potter2 sbin]$ jps
  2. 6464 Master
  3. 6528 Worker
  4. 6561 Jps
  5. 6562 Jps
  6. 3909 NameNode
  7. 6565 Jps
  8. 3703 QuorumPeerMain
  9. 5047 NodeManager
  10. 4412 DFSZKFailoverController
  11. 4204 JournalNode
  12. 4014 DataNode
  1. [potter@potter3 conf]$ jps
  2. 4609 Jps
  3. 3441 DataNode
  4. 3284 QuorumPeerMain
  5. 4581 Worker
  6. 3879 NodeManager
  7. 3576 JournalNode
  8. 3372 NameNode
  9. 3676 DFSZKFailoverController
  1. [potter@potter4 conf]$ jps
  2. 3456 JournalNode
  3. 3607 NodeManager
  4. 4123 Jps
  5. 3356 DataNode
  6. 3260 QuorumPeerMain
  7. 4095 Worker
  1. [potter@potter5 conf]$ jps
  2. 3216 QuorumPeerMain
  3. 3447 NodeManager
  4. 3304 DataNode
  5. 3945 Jps
  6. 3916 Worker

3.5、启动spark

出现以下就算启动成功：

3.5、问题

查看进程发现spark集群只有hadoop1成功启动了Master进程，其他3个节点均没有启动成功，需要手动启动，进入到/home/hadoop/apps/spark/sbin目录下执行以下命令，3个节点都要执行

  1. [potter@potter3 ~]$ cd apps/spark-2.3.0-bin-hadoop2.7/sbin/
  2. [potter@potter3 sbin]$ ./start-master.sh
  1. [potter@potter4 ~]$ cd apps/spark-2.3.0-bin-hadoop2.7/sbin/
  2. [potter@potter4 sbin]$ ./start-master.sh
  1. [potter@potter5 ~]$ cd apps/spark-2.3.0-bin-hadoop2.7/sbin/
  2. [potter@potter5 sbin]$ ./start-master.sh

3.6、执行之后再次查看进程

Master进程和Worker进程都可以启动成功

  1. [potter@potter3 sbin]$ jps
  2. 3441 DataNode
  3. 3284 QuorumPeerMain
  4. 4581 Worker
  5. 3576 JournalNode
  6. 3372 NameNode
  7. 3676 DFSZKFailoverController
  8. 5020 Jps
  9. 4894 Master
  1. [potter@potter4 sbin]$ jps
  2. 3456 JournalNode
  3. 4179 Master
  4. 3356 DataNode
  5. 3260 QuorumPeerMain
  6. 4238 Jps
  7. 4095 Worker
  1. [potter@potter5 sbin]$ jps
  2. 3216 QuorumPeerMain
  3. 3304 DataNode
  4. 4057 Jps
  5. 3916 Worker
  6. 3998 Master

4、验证

4.1、查看Web界面Master状态

potter2是ALIVE状态，potter3、potter4和potter5均是STANDBY状态

potter2

potter3

potter4

potter5

4.2、手动干掉potter2上面的Master进程，观察是否进行自动切换

  1. [potter@potter2 ~]$ jps
  2. 6464 Master
  3. 6528 Worker
  4. 6901 Jps
  5. 3909 NameNode
  6. 3703 QuorumPeerMain
  7. 4412 DFSZKFailoverController
  8. 4204 JournalNode
  9. 4014 DataNode
  10. [potter@potter2 ~]$ kill -9 6464
  11. [potter@potter2 ~]$

干掉potter2上的Master进程之后，再次查看web界面

potter2节点，由于Master进程被干掉，所以界面无法访问

potter3节点，Master被干掉之后，potter3节点上的Master成功篡位成功，成为ALIVE状态

potter4


potter5

五、执行Spark程序on standalone

（1）执行第一个Spark程序
  1. [potter@potter2 apps]$ cd
  2. [potter@potter2 ~]$ /home/potter/apps/spark-2.3.0-bin-hadoop2.7/bin/spark-submit </div>
  3. > –class org.apache.spark.examples.SparkPi </span>
  4. > –master spark?/potter2:7077 </span>
  5. > –executor-memory 500m </span>
  6. > –total-executor-cores 1 </span>
  7. > /home/potter/apps/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar </span>
  8. > 100

其中的spark://hadoop1:7077是下图中的地址

运行结果：

（2）启动spark-shell

  1. /home/potter/apps/spark-2.3.0-bin-hadoop2.7/bin/spark-shell </div>
  2. –master spark://potter2:7077 </span>
  3. –executor-memory 500m </div>
  4. –total-executor-cores 1

参数说明：

–master spark://potter2:7077 指定Master的地址
–executor-memory 500m:指定每个worker可用内存为500m
–total-executor-cores 1: 指定整个集群使用的cup核数为1个

注意：

如果启动spark shell时没有指定master地址，但是也可以正常启动spark shell和执行spark shell中的程序，其实是启动了spark的local模式，该模式仅在本机启动一个进程，没有与集群建立联系。

Spark Shell中已经默认将SparkContext类初始化为对象sc。用户代码如果需要用到，则直接应用sc即可

Spark Shell中已经默认将SparkSQl类初始化为对象spark。用户代码如果需要用到，则直接应用spark即可

（3）在spark shell中编写workcount程序
1）编写一个workcount.txt文件并上传到HDFS上的 / 目录下
  1. [potter@potter2 ~]$ vi workcount.txt
  2. [potter@potter2 ~]$ hadoop fs -put hello.txt /

workcount.txt的内容是：

  1. you，jump
  2. i,jump
2）在spark shell中用scala语言编写spark程序
scala> sc.textFile(”/workcount.txt").flatMap(.split(",")).map((,1)).reduceByKey(+).collect

说明：

sc是SparkContext对象，该对象是提交spark程序的入口
textFile("/spark/hello.txt")是hdfs中读取数据
flatMap(.split(" "))先map再压平
map((,1))将单词和1构成元组
reduceByKey(+)按照key进行reduce，并将value累加
saveAsTextFile("/spark/out")将结果写入到hdfs中

运行结果：

六、执行Spark程序on YARN

（1）前提

成功启动zookeeper集群、HDFS集群、YARN集群

（2）启动Spark on YARN
[potter@potter2 ~]$ spark-shell --master yarn --deploy-mode client

报错如下：

报错原因：内存资源给的过小，yarn直接kill掉进程，则报rpc连接失败、ClosedChannelException等错误。

解决方法：

先停止YARN服务，然后修改yarn-site.xml，增加如下内容

  1. <property>
  2. <name>yarn.nodemanager.vmem-check-enabled</name>
  3. <value>false</value>
  4. <description>Whether virtual memory limits will be enforced for containers</description>
  5. </property>
  6. <property>
  7. <name>yarn.nodemanager.vmem-pmem-ratio</name>
  8. <value>4</value>
  9. <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
  10. </property>

将新的yarn-site.xml文件分发到其他Hadoop节点对应的目录下，最后在重新启动YARN。

重新执行以下命令启动spark on yarn

[potter@potter2 hadoop]$ spark-shell --master yarn --deploy-mode client

（3）打开YARN的web界面

打开YARN的web界面：http：//potter4:8088

可以看到Spark shell应用程序正在运行

单击ID号链接，可以看到该应用程序的详细信息

单击“ApplicationMaster”链接

（4）运行程序
  1. scala> val array = Array(1,2,3,4,5)
  2. array: Array[Int] = Array(1, 2, 3, 4, 5)
  3. scala> val rdd = sc.makeRDD(array)
  4. rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at makeRDD at <console>:26
  5. scala> rdd.count
  6. res0: Long = 5
  7. scala>

再次查看YARN的web界面

（5）执行Spark自带的示例程序PI
  1. spark-submit –class org.apache.spark.examples.SparkPi </span>
  2. –master yarn </span>
  3. –deploy-mode cluster </span>
  4. –driver-memory 500m </span>
  5. –executor-memory 500m </span>
  6. –executor-cores 1 </span>
  7. /home/potter/apps/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar </span>
  8. 10

执行过程：

  1. 2018-04-22 18:16:08 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
  2. 2018-04-22 18:16:13 INFO Client:54 - Requesting a new application from cluster with 4 NodeManagers
  3. 2018-04-22 18:16:14 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
  4. 2018-04-22 18:16:14 INFO Client:54 - Will allocate AM container, with 884 MB memory including 384 MB overhead
  5. 2018-04-22 18:16:14 INFO Client:54 - Setting up container launch context for our AM
  6. 2018-04-22 18:16:14 INFO Client:54 - Setting up the launch environment for our AM container
  7. 2018-04-22 18:16:14 INFO Client:54 - Preparing resources for our AM container
  8. 2018-04-22 18:16:18 WARN Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
  9. 2018-04-22 18:16:26 INFO Client:54 - Uploading resource file:/tmp/spark-e645ee0d-099c-4b22-8729-cb77babf5e0a/__spark_libs__3299474380368175903.zip -> hdfs://myha01/user/potter/.sparkStaging/application_1524389838076_0006/__spark_libs__3299474380368175903.zip
  10. 2018-04-22 18:17:04 INFO Client:54 - Uploading resource file:/home/potter/apps/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar -> hdfs://myha01/user/potter/.sparkStaging/application_1524389838076_0006/spark-examples_2.11-2.3.0.jar
  11. 2018-04-22 18:17:05 INFO Client:54 - Uploading resource file:/tmp/spark-e645ee0d-099c-4b22-8729-cb77babf5e0a/__spark_conf__7169638864757569614.zip -> hdfs://myha01/user/potter/.sparkStaging/application_1524389838076_0006/spark_conf.zip
  12. 2018-04-22 18:17:05 INFO SecurityManager:54 - Changing view acls to: potter
  13. 2018-04-22 18:17:05 INFO SecurityManager:54 - Changing modify acls to: potter
  14. 2018-04-22 18:17:05 INFO SecurityManager:54 - Changing view acls groups to:
  15. 2018-04-22 18:17:05 INFO SecurityManager:54 - Changing modify acls groups to:
  16. 2018-04-22 18:17:05 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(potter); groups with view permissions: Set(); users with modify permissions: Set(potter); groups with modify permissions: Set()
  17. 2018-04-22 18:17:06 INFO Client:54 - Submitting application application_1524389838076_0006 to ResourceManager
  18. 2018-04-22 18:17:06 INFO YarnClientImpl:273 - Submitted application application_1524389838076_0006
  19. 2018-04-22 18:17:07 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  20. 2018-04-22 18:17:07 INFO Client:54 -
  21. client token: N/A
  22. diagnostics: N/A
  23. ApplicationMaster host: N/A
  24. ApplicationMaster RPC port: -1
  25. queue: default
  26. start time: 1524392325362
  27. final status: UNDEFINED
  28. tracking URL: http://potter4:8088/proxy/application_1524389838076_0006/
  29. user: potter
  30. 2018-04-22 18:17:08 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  31. 2018-04-22 18:17:09 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  32. 2018-04-22 18:17:10 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  33. 2018-04-22 18:17:11 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  34. 2018-04-22 18:17:12 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  35. 2018-04-22 18:17:13 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  36. 2018-04-22 18:17:14 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  37. 2018-04-22 18:17:15 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  38. 2018-04-22 18:17:16 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  39. 2018-04-22 18:17:17 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  40. 2018-04-22 18:17:18 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  41. 2018-04-22 18:17:19 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  42. 2018-04-22 18:17:20 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  43. 2018-04-22 18:17:22 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  44. 2018-04-22 18:17:23 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  45. 2018-04-22 18:17:24 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  46. 2018-04-22 18:17:25 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  47. 2018-04-22 18:17:26 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  48. 2018-04-22 18:17:28 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  49. 2018-04-22 18:17:29 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  50. 2018-04-22 18:17:30 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  51. 2018-04-22 18:17:31 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  52. 2018-04-22 18:17:33 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  53. 2018-04-22 18:17:34 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  54. 2018-04-22 18:17:35 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  55. 2018-04-22 18:17:36 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  56. 2018-04-22 18:17:37 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  57. 2018-04-22 18:17:38 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  58. 2018-04-22 18:17:40 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  59. 2018-04-22 18:17:41 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  60. 2018-04-22 18:17:42 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  61. 2018-04-22 18:17:43 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  62. 2018-04-22 18:17:44 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  63. 2018-04-22 18:17:45 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  64. 2018-04-22 18:17:46 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  65. 2018-04-22 18:17:47 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  66. 2018-04-22 18:17:48 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  67. 2018-04-22 18:17:49 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  68. 2018-04-22 18:17:50 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  69. 2018-04-22 18:17:51 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  70. 2018-04-22 18:17:52 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  71. 2018-04-22 18:17:53 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  72. 2018-04-22 18:17:54 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  73. 2018-04-22 18:17:55 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  74. 2018-04-22 18:17:56 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  75. 2018-04-22 18:17:57 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  76. 2018-04-22 18:17:58 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  77. 2018-04-22 18:17:59 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  78. 2018-04-22 18:18:00 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  79. 2018-04-22 18:18:01 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  80. 2018-04-22 18:18:02 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  81. 2018-04-22 18:18:03 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  82. 2018-04-22 18:18:04 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  83. 2018-04-22 18:18:05 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  84. 2018-04-22 18:18:06 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  85. 2018-04-22 18:18:07 INFO Client:54 - Application report for application_1524389838076_0006 (state: ACCEPTED)
  86. 2018-04-22 18:18:08 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  87. 2018-04-22 18:18:08 INFO Client:54 -
  88. client token: N/A
  89. diagnostics: N/A
  90. ApplicationMaster host: 192.168.123.102
  91. ApplicationMaster RPC port: 0
  92. queue: default
  93. start time: 1524392325362
  94. final status: UNDEFINED
  95. tracking URL: http://potter4:8088/proxy/application_1524389838076_0006/
  96. user: potter
  97. 2018-04-22 18:18:09 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  98. 2018-04-22 18:18:10 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  99. 2018-04-22 18:18:11 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  100. 2018-04-22 18:18:12 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  101. 2018-04-22 18:18:13 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  102. 2018-04-22 18:18:14 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  103. 2018-04-22 18:18:15 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  104. 2018-04-22 18:18:16 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  105. 2018-04-22 18:18:17 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  106. 2018-04-22 18:18:19 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  107. 2018-04-22 18:18:20 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  108. 2018-04-22 18:18:21 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  109. 2018-04-22 18:18:23 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  110. 2018-04-22 18:18:24 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  111. 2018-04-22 18:18:25 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  112. 2018-04-22 18:18:26 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  113. 2018-04-22 18:18:27 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  114. 2018-04-22 18:18:28 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  115. 2018-04-22 18:18:29 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  116. 2018-04-22 18:18:30 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  117. 2018-04-22 18:18:31 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  118. 2018-04-22 18:18:32 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  119. 2018-04-22 18:18:33 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  120. 2018-04-22 18:18:35 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  121. 2018-04-22 18:18:36 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  122. 2018-04-22 18:18:37 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  123. 2018-04-22 18:18:38 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  124. 2018-04-22 18:18:39 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  125. 2018-04-22 18:18:40 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  126. 2018-04-22 18:18:41 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  127. 2018-04-22 18:18:42 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  128. 2018-04-22 18:18:43 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  129. 2018-04-22 18:18:44 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  130. 2018-04-22 18:18:45 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  131. 2018-04-22 18:18:46 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  132. 2018-04-22 18:18:47 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  133. 2018-04-22 18:18:48 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  134. 2018-04-22 18:18:49 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  135. 2018-04-22 18:18:50 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  136. 2018-04-22 18:18:51 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  137. 2018-04-22 18:18:52 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  138. 2018-04-22 18:18:53 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  139. 2018-04-22 18:18:54 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  140. 2018-04-22 18:18:55 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  141. 2018-04-22 18:18:56 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  142. 2018-04-22 18:18:57 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  143. 2018-04-22 18:18:58 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  144. 2018-04-22 18:18:59 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  145. 2018-04-22 18:19:00 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  146. 2018-04-22 18:19:01 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  147. 2018-04-22 18:19:02 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  148. 2018-04-22 18:19:03 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  149. 2018-04-22 18:19:04 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  150. 2018-04-22 18:19:05 INFO Client:54 - Application report for application_1524389838076_0006 (state: RUNNING)
  151. 2018-04-22 18:19:06 INFO Client:54 - Application report for application_1524389838076_0006 (state: FINISHED)
  152. 2018-04-22 18:19:06 INFO Client:54 -
  153. client token: N/A
  154. diagnostics: N/A
  155. ApplicationMaster host: 192.168.123.102
  156. ApplicationMaster RPC port: 0
  157. queue: default
  158. start time: 1524392325362
  159. final status: SUCCEEDED
  160. tracking URL: http://potter4:8088/proxy/application_1524389838076_0006/
  161. user: potter
  162. 2018-04-22 18:19:08 INFO ShutdownHookManager:54 - Shutdown hook called
  163. 2018-04-22 18:19:09 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-6f009c18-9d50-460b-b480-77b0ca856369
  164. 2018-04-22 18:19:09 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-e645ee0d-099c-4b22-8729-cb77babf5e0a
  165. [potter@potter2 ~]$

Spark学习（二）Spark高可用集群搭建相关推荐

  1. .Net Core2.1 秒杀项目一步步实现CI/CD(Centos7)系列二:k8s高可用集群搭建总结以及部署API到k8s...

    前言:本系列博客又更新了,是博主研究很长时间,亲自动手实践过后的心得,k8s集群是购买了5台阿里云服务器部署的,这个集群差不多搞了一周时间,关于k8s的知识点,我也是刚入门,这方面的知识建议参考博客园 ...

  2. mycat mysql好可用架构_想要学会MyCat高可用集群搭建，但是这些知识点却还玩不明白？...

    一.集群架构 1.MyCat实现读写分离架构 在我前面的文章, 我已经讲解过了通过MyCat来实现MySQL的读写分离, 从而完成MySQL集群的负载均衡 , 如下面的结构图: 但是以上架构存在问题 ...

  3. Hadoop(二): YARN(资源管理器 RM)、HBase高可用集群搭建

    目录 一.Hadoop的高可用原理概述 1.1 原理概述 1.2 实验环境要求 二.高可用集群的搭建 2.1 搭建Zookeeper 2.2 搭建HA的高可用 2.3 YARN(资源管理器 RM)高可 ...

  4. Hadoop 3.1.2(HA)+Zookeeper3.4.13+Hbase1.4.9(HA)+Hive2.3.4+Spark2.4.0(HA)高可用集群搭建

    目录 目录 1.前言 1.1.什么是 Hadoop? 1.1.1.什么是 YARN? 1.2.什么是 Zookeeper? 1.3.什么是 Hbase? 1.4.什么是 Hive 1.5.什么是 Sp ...

  5. RabbitMQ高级指南：从配置、使用到高可用集群搭建

    本文大纲: 1. RabbitMQ简介 2. RabbitMQ安装与配置 3. C# 如何使用RabbitMQ 4. 几种Exchange模式 5. RPC 远程过程调用 6. RabbitMQ高可用 ...

  6. RabbitMQ高可用集群搭建

    RabbitMQ高可用集群搭建 摘要:实际生产应用中都会采用消息队列的集群方案,如果选择RabbitMQ那么有必要了解下它的集群方案原理一般来说,如果只是为了学习RabbitMQ或者验证业务工程的正确 ...

  7. Hadoop HA 高可用集群搭建

    Hadoop HA 高可用集群搭建 一.首先配置集群信息 1 vi /etc/hosts 二.安装zookeeper 1.解压至/usr/hadoop/下 1 tar -zxvf zookeeper- ...

  8. Hadoop HA高可用集群搭建（Hadoop+Zookeeper+HBase）

    一.服务器环境 主机名 IP 用户名 密码 安装目录 master 192.168.142.124 root xxx /usr/hadoop-2.6.5/ slave1 192.168.142.125 ...

  9. k8s高可用集群搭建部署

    简介 k8s普通搭建出来只是单master节点,如果该节点挂掉,则整个集群都无法调度,K8s高可用集群是用多个master节点加负载均衡节点组成,外层再接高可用分布式存储集群例如ceph集群,实现计算 ...

  10. RabbitMQ 高级指南：从配置、使用到高可用集群搭建

    博主说:在项目中,通过 RabbitMQ,咱们可以将一些无需即时返回且耗时的操作提取出来,进行异步处理,而这种异步处理的方式大大的节省了服务器的请求响应时间,从而提高了系统的吞吐量. 正文 1 Rab ...

最新文章

  1. python肘部法则 最优分类
  2. matlab调用opencv的函数
  3. 更改记录表CDHDR和CDPOS
  4. VTK：Snippets之RestoreSceneFromFieldData
  5. 《Inside XAML》翻译半成品
  6. 开源代码分析技巧之——打印调用逻辑
  7. Xshell6突然连不上K8S所在的虚拟机
  8. 基于python的智能风扇设计_智能风扇设计毕业设计
  9. idea在创建类时在File Header中加入昵称和时间等
  10. JS实现文本全选并复制
  11. 新常态 新核心，浪潮商用机器为关键行业数字化转型打造新Power
  12. Linux_数据段、代码段、堆栈段、BSS段的区别
  13. 微博java敏感词_新浪微博的敏感词是哪些？
  14. 微信小程序页面的基本布局方法——flex布局
  15. Golang--Go语言 五百行后台代码实现一简约的个人博客网站-TinyBlog
  16. TMC2208电机驱动简介
  17. linux上读取不到库文件,linux中make找不到库文件-lmpi的问题
  18. nvm的安装配置教程
  19. subtract用法c语言,操作 subtract() - 闪电教程JSRUN
  20. java 删除语句_是java语句

热门文章

  1. 【十五、网站公安备案】2021最详细wordpress博客建站教程（2021.03.04更新）
  2. Cygwin的替代软件Gow
  3. 阿里P8架构师谈(9)：流量高峰时期的性能瓶颈有哪些、以及如何来解决
  4. 复制加密内存卡(TF卡、U盘)资料的方法
  5. matlab求解常微分方程,matlab 求解常微分方程式
  6. python在房地产中的应用_“人生苦短，我学 Python”丨爆火的Python语言应用领域主要有哪些？...
  7. git push 失败与解决方法
  8. DOM操作 （创建、增、删、改、查、属性操作、事件操作）
  9. 向U盘拷贝文件，总是提示对于目标系统，文件过大？？
  10. catboost案例