flink配置historyserver

flink版本1.9.1

1.编辑flink/conf/flink-conf.yaml ，搜索history，主要编辑如下

#==============================================================================
# HistoryServer
#==============================================================================# The HistoryServer is started and stopped via bin/historyserver.sh (start|stop)
#
# # Directory to upload completed jobs to. Add this directory to the list of
# # monitored directories of the HistoryServer as well (see below).
# #jobmanager.archive.fs.dir: hdfs:///completed-jobs/jobmanager.archive.fs.dir: hdfs://beh/flink/
#
# # The address under which the web-based HistoryServer listens.
# #historyserver.web.address: 0.0.0.0historyserver.web.address: 10.198.162.24
#
# # The port under which the web-based HistoryServer listens.
# #historyserver.web.port: 8082historyserver.web.port: 8083# # Comma separated list of directories to monitor for completed jobs.
# #historyserver.archive.fs.dir: hdfs:///completed-jobs/historyserver.archive.fs.dir: hdfs://beh/flink/# Interval in milliseconds for refreshing the monitored directories.
#historyserver.archive.fs.refresh-interval: 10000
historyserver.archive.fs.refresh-interval: 10000

2.参数释义

jobmanager.archive.fs.dir：flink job运行完成后的日志存放目录
historyserver.archive.fs.dir：flink history进程的hdfs监控目录
historyserver.web.address：flink history进程所在的主机
historyserver.web.port：flink history进程的占用端口
historyserver.archive.fs.refresh-interval：刷新受监视目录的时间间隔（以毫秒为单位）。

3.注意jobmanager.archive.fs.dir要和historyserver.archive.fs.dir值一样：

实例：jobmanager.archive.fs.dir:为hdfs://beh/flink/

historyserver.archive.fs.dir:为hdfs://beh/flink/jobs

job跑完出现如下：

然后查看history页面发现内容为空。

修改 historyserver.archive.fs.dir:为hdfs://beh/flink/后，再次运行job后如下：

然后查看history页面发现不仅吸现在的job，之前运行的也显示出来了。

注1：可以把其他集群的jobs日志拉过来，然后启动history进程进行页面展示。

注2：启动过程可能报错：

2019-12-20 15:55:29,985 WARN  org.apache.flink.runtime.webmonitor.history.HistoryServer     - Failed to create Path or FileSystem fo
r directory 'hdfs://beh/flink/'. Directory will not be monitored.
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The sc
heme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:447)at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:359)at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:174)at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:137)at org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:122)at org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:119)at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)at org.apache.flink.runtime.webmonitor.history.HistoryServer.main(HistoryServer.java:119)
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.at org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:58)at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:443)... 8 more
2019-12-20 15:55:29,988 ERROR org.apache.flink.runtime.webmonitor.history.HistoryServer     - Failed to run HistoryServer.
org.apache.flink.util.FlinkException: Failed to validate any of the configured directories to monitor.at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:183)at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:137)at org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:122)

具体解决办法见：https://my.oschina.net/u/2338224/blog/3101005（敬佩大佬无私）

原因是在flink集群的CLASS_PATH下缺少了 HDFS相关的jar，其下载地址为（看个人hadoop和flink版本）：

https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.6.5-9.0/

flink配置historyserver相关推荐

聊聊flink的HistoryServer
为什么80%的码农都做不了架构师?>>> 序本文主要研究一下flink的HistoryServer HistoryServer flink-1.7.2/flink-runti ...
flink配置+停止flink中的job
首先是下载哪个压缩包的问题: [2]是pyflink,但是根据官方说目前属于支持很初始的情况. 所以我们来下载[3] 配置文件配置 $FLINK_HOME/conf/flink-conf.yaml ...
配置HistoryServer
一: 默认情况下HistoryServer没有启动,需要使用sbin/mr-jobhistory-daemon.sh start historyserver 启动二: 启动前需要在mapred-si ...
flink 高可用 high-availability 配置的重试次数无效问题
问题背景我根据这篇博客https://blog.csdn.net/cndotaci/article/details/106870413的介绍,配置了flink基于yarn的高可用,测试时发现配置的任 ...
kylin如何支持flink_大数据集群运维（28） Kylin 配置使用 flink 构建 Cube
用 Flink 构建 Cube Kylin v3.1 引入了 Flink cube engine,在 build cube 步骤中使用 Apache Flink 代替 MapReduce:您可以查看 ...
Flink on Yarn（HA配置）
根据部署方式不同,Flink Jobmanager HA配置分为2种: 1.standalone cluster HA 2.Yarn cluster HA 其中,standalone cluster ...
Flink HA配置
JobManager 高可用(HA) jobManager协调每个flink任务部署.它负责调度和资源管理. 默认情况下,每个flink集群只有一个JobManager,这将导致一个单点故障(SPOF ...
[基础架构] [Flink] Flink/Flink-CDC的部署和配置
简介下载官方Flink依赖包 (笔者所用版本为1.13.6) 下载下面列出的依赖包,并将它们放到目录 flink-1.13.6/lib/ 下: 下载elasticsearch连接器flink-sql ...
Linux下安装配置各种软件和服务
1. JDK 1.1. 解压Linux版本的JDK压缩包 yum -y install glibc.i686 #安装jdk源glibc(需要联网下载源) mkdir /usr/local/src/jd ...

flink配置historyserver

flink配置historyserver相关推荐

最新文章

热门文章