生产环境实战spark （10）分布式集群 5台设备 SPARK集群 HistoryServer WEBUI不能打开问题解决 File file:/tmp/spark-events does not

生产环境实战spark （10）分布式集群 5台设备 SPARK集群 HistoryServer WEBUI不能打开问题解决

这个是个老问题，之前也遇到过，启动start-history-server.sh，报错如下

Caused by: java.io.FileNotFoundException: Log directory specified does not exist: file:/tmp/spark-events Did you configure the correct one through spark.history.fs.logDirectory?

[root@master sbin]# start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to /usr/local/spark-2.1.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.history.HistoryServer-1-master.out
failed to launch: nice -n 0 /usr/local/spark-2.1.0-bin-hadoop2.6/bin/spark-class org.apache.spark.deploy.history.HistoryServerat org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:197)... 9 more
full log in /usr/local/spark-2.1.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.history.HistoryServer-1-master.out[root@master sbin]# cat /usr/local/spark-2.1.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.history.HistoryServer-1-master.out

Spark Command: /usr/local/jdk1.8.0_121/bin/java -cp /usr/local/spark-2.1.0-bin-hadoop2.6/conf/:/usr/local/spark-2.1.0-bin-hadoop2.6/jars/*:/usr/local/hadoop-2.6.5/etc/hadoop/ -Xmx1g org.apache.spark.deploy.history.HistoryServer
========================================
17/04/27 12:15:27 INFO history.HistoryServer: Started daemon with process name: 31814@master
17/04/27 12:15:27 INFO util.SignalUtils: Registered signal handler for TERM
17/04/27 12:15:27 INFO util.SignalUtils: Registered signal handler for HUP
17/04/27 12:15:27 INFO util.SignalUtils: Registered signal handler for INT
17/04/27 12:15:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/27 12:15:28 INFO spark.SecurityManager: Changing view acls to: root
17/04/27 12:15:28 INFO spark.SecurityManager: Changing modify acls to: root
17/04/27 12:15:28 INFO spark.SecurityManager: Changing view acls groups to:
17/04/27 12:15:28 INFO spark.SecurityManager: Changing modify acls groups to:
17/04/27 12:15:28 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
Exception in thread "main" java.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:278)at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.io.FileNotFoundException: Log directory specified does not exist: file:/tmp/spark-events Did you configure the correct one through spark.history.fs.logDirectory?at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:207)at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:153)at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:149)at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:77)... 6 more
Caused by: java.io.FileNotFoundException: File file:/tmp/spark-events does not existat org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:537)at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:750)at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:527)at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:197)... 9 more

问题解决：

1，修改spark的配置文件spark-defaults.conf

加上spark.history.fs.logDirectory hdfs://Master:9000/historyserverforSpark

spark.eventLog.enabled false

[root@master conf]# ls
docker.properties.template  fairscheduler.xml.template  log4j.properties.template  metrics.properties.template  slaves  spark-defaults.conf  spark-env.sh
[root@master conf]# vi  spark-defaults.conf # Default system properties included when running spark-submit.
# This is useful for setting default environmental settings.# Example:
# spark.master                     spark://master:7077spark.eventLog.enabled           false
# spark.eventLog.dir               hdfs://namenode:8021/directory
# spark.serializer                 org.apache.spark.serializer.KryoSerializer
# spark.driver.memory              5g
# spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"spark.history.fs.logDirectory    hdfs://Master:9000/historyserverforSpark

2，将spark-defaults.conf 文件分发到各个worker节点

[root@master conf]# ssh   root@10.*.*.238  rm -rf /usr/local/spark-2.1.0-bin-hadoop2.6/conf/
[root@master conf]# ssh   root@10.*.*.239  rm -rf /usr/local/spark-2.1.0-bin-hadoop2.6/conf/
[root@master conf]# ssh   root@10.*.*.240  rm -rf /usr/local/spark-2.1.0-bin-hadoop2.6/conf/
[root@master conf]# ssh   root@10.*.*.241  rm -rf /usr/local/spark-2.1.0-bin-hadoop2.6/conf/
[root@master conf]# scp   -rq /usr/local/spark-2.1.0-bin-hadoop2.6/conf root@10.*.*.238:/usr/local/spark-2.1.0-bin-hadoop2.6
scp   -rq /usr/local/spark-2.1.0-bin-hadoop2.6/conf root@10.*.*.239:/usr/local/spark-2.1.0-bin-hadoop2.6
scp   -rq /usr/local/spark-2.1.0-bin-hadoop2.6/conf root@10.*.*.240:/usr/local/spark-2.1.0-bin-hadoop2.6
scp   -rq /usr/local/spark-2.1.0-bin-hadoop2.6/conf root@10.*.*.241:/usr/local/spark-2.1.0-bin-hadoop2.6
[root@master conf]# scp   -rq /usr/local/spark-2.1.0-bin-hadoop2.6/conf root@10.*.*.239:/usr/local/spark-2.1.0-bin-hadoop2.6
[root@master conf]# scp   -rq /usr/local/spark-2.1.0-bin-hadoop2.6/conf root@10.*.*.240:/usr/local/spark-2.1.0-bin-hadoop2.6
[root@master conf]# scp   -rq /usr/local/spark-2.1.0-bin-hadoop2.6/conf root@10.*.*.241:/usr/local/spark-2.1.0-bin-hadoop2.6
[root@master conf]#

3，在hdfs中建立 hdfs://Master:9000/historyserverforSpark目录

[root@master sbin]# hdfs dfs -ls /
Found 1 items
drwx-wx-wx   - root supergroup          0 2017-04-25 16:08 /tmp
[root@master sbin]# hdfs dfs  -mkdir historyserverforSpark
mkdir: `historyserverforSpark': No such file or directory
[root@master sbin]# hdfs dfs  -mkdir  hdfs://Master:9000/historyserverforSpark
[root@master sbin]# hdfs dfs -ls /
Found 2 items
drwxr-xr-x   - root supergroup          0 2017-04-27 12:23 /historyserverforSpark
drwx-wx-wx   - root supergroup          0 2017-04-25 16:08 /tmp
[root@master sbin]# jps

4，重启启动spark的start-history-server.sh，成功启动

[root@master sbin]# start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to /usr/local/spark-2.1.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.history.HistoryServer-1-master.out
[root@master sbin]#

5，查看页面

生产环境实战spark （10）分布式集群 5台设备 SPARK集群 HistoryServer WEBUI不能打开问题解决 File file:/tmp/spark-events does not相关推荐

生产环境实战spark （5）分布式集群 5台设备之间hosts文件配置 ssh免密码登录
生产环境实战spark (5)分布式集群 5台设备之间 ssh免密码登录之前已经在master节点单台设备上配置ssh免密码的登录工作,现在要做的事情是在5台设备之间实现ssh免密码操作.我在整个大 ...
【kafka实战】分区重分配可能出现的问题和排查问题思路(生产环境实战,附视频)
超强!!! Kafka高质量专栏学习大全,点我获取!!! 文章目录前提所有异常情况 1. TargetBroker若不在线,迁移脚本执行会失败情景演示 2. TargetBroker在开始迁移过 ...
企业级负载均衡集群——通过fence设备解决集群节点之间争抢资源的现象（FENCE搭建、高可用服务配置详解）
1.FENCE工具的原理及作用 FENCE设备是RHCS集群中必不可少的一个组成部分,通过FENCE设备可以避免因出现不可预知的情况而造成的"脑裂"现象 FENCE设备的出现,就是 ...
前端H5项目部署到OSS-利用jenkins实现自动发布【生产环境实战】
前端H5项目发布到OSS 文章目录前端H5项目发布到OSS 背景一.创建Bucket 二.为Bucket绑定自定义域名 1.购买的域名和oss在同一个阿里云账号下(大多数) 2.购买的域名和oss ...
SpingBoot+Quartrz生产环境的应用支持分布式、自定义corn、反射执行多任务
生产环境可以用吗_小型熔喷布设备可以生产出好的熔喷布吗？
原标题:小型熔喷布设备可以生产出好的熔喷布吗? 一般中小型的生产流水线主要是:45/50/65型挤塑机,中小型料管,及其一般熔喷化学纤维料.有朋友就说了,像一些小型的熔喷布生产设备能生产出合格的熔喷布 ...
新年上班第一天生产环境分布式文件系统崩了！！
写在前面说来也怪,早不崩晚不崩,偏偏在上班第一天的时候,生产环境分布式文件系统崩了.我才刚来到我的工位坐下,"叮铃铃"电话响了,是运营打来的,"喂,冰河,快点看看,生产 ...
Spark基础学习笔记04：搭建Spark伪分布式环境
文章目录零.本讲学习目标一.搭建伪分布式Hadoop (一)登录ied虚拟机 (二)配置免密登录 1.生成密钥对 2.将生成的公钥发送到本机(虚拟机ied) 3.验证虚拟机是否能免密登录本机 (三 ...
ClickHouse - 多卷存储扩大存储容量（生产环境必备）
笔者最近工作有点忙,加上培训较多,近期文章更新慢了一拍.不过,今天为 ClickHouse 的爱好者带来一篇非常不错的文章,部分内容来自 Altinity,以及笔者补充和整理而成. 长期使用 Clic ...
Nacos 发布 1.0.0 GA 版本，可大规模投入到生产环境
经过 3 个 RC 版本的社区体验之后,Nacos 正式发布 1.0.0 GA 版本,在架构.功能和 API 设计上进行了全方位的重构和升级. 1.0.0 版本的发布标志着 Nacos 已经可以大规模 ...