背景:

项目需要在spark任务提交服务器节点上自动提交任务到spark集群上。因此创建了一个固定时间监控任务项目,使用timer定时监控oracle数据库中是否有spark提交任务,如果有spark提交任务,就使用java.lang.ProcessBuilder对象调sh文件(spark-submit.sh),来提交任务。

部署项目、wrapper x64到CentOS服务器(spark任务提交服务器)

1、首先需要下载wrapper x64 linux版本上的安装包:http://pan.baidu.com/s/1nvc7JC1

2、将wrapper x64 linux服务器工具包下载到centos x64服务上,在centos x64上存储路径为:/app/tt/service/wrapper。

[tt@vh01 wrapper]$ ls
bin  conf  doc  lib  logs  src

3、把使用timer实现的固定时间间隔执行监控提交spark任务的java项目发布成My_AutoExecutePlan.jar包,上传到/app/tt/service/wrapper/lib下,并把其依赖的包也上传到/app/tt/service/wrapper/lib下

[tt@vh01 wrapper]$ cd lib
[tt@vh01 lib]$ ls
commons-logging.jar  fastjson-1.2.10.jar           libwrapper.so  ojdbc7.jar       wrapper.jar
downjar.sh           My_AutoExecutePlan.jar  log4j.jar      wrapperdemo.jar  wrappertest.jar

备注:

  1. 其中标记为灰色的jar包为wrapper默认的自带的jar包,不可以删除;
  2. 其中标记为红色的jar包为固定时间监控的jar包和其所以来的jar包;
  3. 其中标记为蓝色的sh为下载文件的shell命令文件。

4、进入到/app/tt/service/wrapper/bin下,创建script文件夹,把需要spark-submit.sh文件放到/app/tt/service/wrapper/bin/script路径下;创建conf文件把log4.properties、jdbc.properties等文件存放到/app/tt/service/wrapper/bin/conf下;把spark-submit.sh所需要的spark任务包ParserLogOnSpark.jar包存放到/app/tt/service/wrapper/bin下:

[tt@vh01 wrapper]$ cd bin
[tt@vh01 bin]$ ls
conf                  scripts
demoapp                 ParserLogOnSpark.jar     wrapper       SparkSubmitService

SparkSubmitService文件为服务器启动文件,它是由demoapp文件复制而来:

# Application
APP_NAME="SparkSubmitService"
APP_LONG_NAME="Task_SparkSubmitService"# If uncommented (and set to false), APP_NAME and APP_LONG_NAME will no longer
# be passed to the wrapper. See documentation for details.
#APP_NAME_PASS_TO_WRAPPER=false# Wrapper
WRAPPER_CMD="./wrapper"
WRAPPER_CONF="../conf/wrapper.conf"

注意:

  1. 该文件是从demoapp文件复制而来,名称只需要有意义就可以;
  2. 文件内容中只需要修改"APP_NAME" & "APP_LONG_NAME"的名称为一个有意义的名称;
  3. 该文件告诉了另外一个重要信息就是真正wrapper配置文件是在/app/tt/service/wrapper/conf下的wrapper.conf

5、配置/app/tt/service/wrapper/conf下的wrapper.conf文件内容:

#encoding=UTF-8
# Configuration files must begin with a line specifying the encoding
#  of the the file.
#
# NOTE - Please use src/conf/wrapper.conf.in as a template for your
#        own application rather than the values used for the
#        TestWrapper sample.#********************************************************************
# Wrapper License Properties (Ignored by Community Edition)
#********************************************************************
# Professional and Standard Editions of the Wrapper require a valid
#  License Key to start.  Licenses can be purchased or a trial license
#  requested on the following pages:
# http://wrapper.tanukisoftware.com/purchase
# http://wrapper.tanukisoftware.com/trial# Include file problems can be debugged by removing the first '#'
#  from the following line:
##include.debug# The Wrapper will look for either of the following optional files for a
#  valid License Key.  License Key properties can optionally be included
#  directly in this configuration file.
#include ../conf/wrapper-license.conf
#include ../conf/wrapper-license-%WRAPPER_HOST_NAME%.conf# The following property will output information about which License Key(s)
#  are being found, and can aid in resolving any licensing problems.
#wrapper.license.debug=TRUE#********************************************************************
# Wrapper Localization
#********************************************************************
# Specify the locale which the Wrapper should use.  By default the system
#  locale is used.
#wrapper.lang=en_US # en_US or ja_JP# Specify the location of the Wrapper's language resources.  If these are
#  missing, the Wrapper will default to the en_US locale.
wrapper.lang.folder=../lang#********************************************************************
# Wrapper Java Properties
#********************************************************************
# Java Application
#  Locate the java binary on the system PATH:
#wrapper.java.command=java
#  Specify a specific java binary:
#set.JAVA_HOME=/java/path
wrapper.java.command=/usr/local/jdk1.8.0_77/bin/java# Tell the Wrapper to log the full generated Java command line.
#wrapper.java.command.loglevel=INFO# Java Main class.  This class must implement the WrapperListener interface
#  or guarantee that the WrapperManager class is initialized.  Helper
#  classes are provided to do this for you.  See the Integration section
#  of the documentation for details.
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp# Java Classpath (include wrapper.jar)  Add class path elements as
#  needed starting from 1
wrapper.java.classpath.1=../lib/My_AutoExecutePlan.jar
wrapper.java.classpath.2=../lib/wrapper.jar
wrapper.java.classpath.3=../lib/commons-logging.jar
wrapper.java.classpath.4=../lib/fastjson-1.2.10.jar
wrapper.java.classpath.5=../lib/log4j.jar
wrapper.java.classpath.6=../lib/ojdbc7.jar
wrapper.java.classpath.7=/usr/local/jdk1.8.0_77/lib/dt.jar
wrapper.java.classpath.8=/usr/local/jdk1.8.0_77/lib/tools.jar# Java Library Path (location of Wrapper.DLL or libwrapper.so)
wrapper.java.library.path.1=../lib# Java Bits.  On applicable platforms, tells the JVM to run in 32 or 64-bit mode.
wrapper.java.additional.auto_bits=TRUE# Java Additional Parameters
wrapper.java.additional.1=# Initial Java Heap Size (in MB)
#wrapper.java.initmemory=3# Maximum Java Heap Size (in MB)
#wrapper.java.maxmemory=64# Application parameters.  Add parameters as needed starting from 1
wrapper.app.parameter.1=My Project Jar Main Class Name#********************************************************************
# Wrapper Logging Properties
#********************************************************************
# Enables Debug output from the Wrapper.
# wrapper.debug=TRUE# Format of output for the console.  (See docs for formats)
wrapper.console.format=PM# Log Level for console output.  (See docs for log levels)
wrapper.console.loglevel=INFO# Log file to use for wrapper output logging.
wrapper.logfile=../logs/wrapper.log# Format of output for the log file.  (See docs for formats)
wrapper.logfile.format=LPTM# Log Level for log file output.  (See docs for log levels)
wrapper.logfile.loglevel=INFO# Maximum size that the log file will be allowed to grow to before
#  the log is rolled. Size is specified in bytes.  The default value
#  of 0, disables log rolling.  May abbreviate with the 'k' (kb) or
#  'm' (mb) suffix.  For example: 10m = 10 megabytes.
wrapper.logfile.maxsize=0# Maximum number of rolled log files which will be allowed before old
#  files are deleted.  The default value of 0 implies no limit.
wrapper.logfile.maxfiles=0# Log Level for sys/event log output.  (See docs for log levels)
wrapper.syslog.loglevel=NONE#********************************************************************
# Wrapper General Properties
#********************************************************************
# Allow for the use of non-contiguous numbered properties
wrapper.ignore_sequence_gaps=TRUE# Do not start if the pid file already exists.
wrapper.pidfile.strict=TRUE# Title to use when running as a console
wrapper.console.title=SparkSubmitService Wrapper Application#********************************************************************
# Wrapper JVM Checks
#********************************************************************
# Detect DeadLocked Threads in the JVM. (Requires Standard Edition)
wrapper.check.deadlock=TRUE
wrapper.check.deadlock.interval=10
wrapper.check.deadlock.action=RESTART
wrapper.check.deadlock.output=FULL# Out Of Memory detection.
# (Ignore output from dumping the configuration to the console.  This is only needed by the TestWrapper sample application.)
wrapper.filter.trigger.999=wrapper.filter.trigger.*java.lang.OutOfMemoryError
wrapper.filter.allow_wildcards.999=TRUE
wrapper.filter.action.999=NONE
#  Ignore -verbose:class output to avoid false positives.
wrapper.filter.trigger.1000=[Loaded java.lang.OutOfMemoryError
wrapper.filter.action.1000=NONE
# (Simple match)
wrapper.filter.trigger.1001=java.lang.OutOfMemoryError
# (Only match text in stack traces if -XX:+PrintClassHistogram is being used.)
#wrapper.filter.trigger.1001=Exception in thread "*" java.lang.OutOfMemoryError
#wrapper.filter.allow_wildcards.1001=TRUE
wrapper.filter.action.1001=RESTART
wrapper.filter.message.1001=The JVM has run out of memory.#********************************************************************
# Wrapper Email Notifications. (Requires Professional Edition)
#********************************************************************
# Common Event Email settings.
#wrapper.event.default.email.debug=TRUE
#wrapper.event.default.email.smtp.host=<SMTP_Host>
#wrapper.event.default.email.smtp.port=25
#wrapper.event.default.email.subject=[%WRAPPER_HOSTNAME%:%WRAPPER_NAME%:%WRAPPER_EVENT_NAME%] Event Notification
#wrapper.event.default.email.sender=<Sender email>
#wrapper.event.default.email.recipient=<Recipient email># Configure the log attached to event emails.
#wrapper.event.default.email.attach_log=TRUE
#wrapper.event.default.email.maillog.lines=50
#wrapper.event.default.email.maillog.format=LPTM
#wrapper.event.default.email.maillog.loglevel=INFO# Enable specific event emails.
#wrapper.event.wrapper_start.email=TRUE
#wrapper.event.jvm_prelaunch.email=TRUE
#wrapper.event.jvm_start.email=TRUE
#wrapper.event.jvm_started.email=TRUE
#wrapper.event.jvm_deadlock.email=TRUE
#wrapper.event.jvm_stop.email=TRUE
#wrapper.event.jvm_stopped.email=TRUE
#wrapper.event.jvm_restart.email=TRUE
#wrapper.event.jvm_failed_invocation.email=TRUE
#wrapper.event.jvm_max_failed_invocations.email=TRUE
#wrapper.event.jvm_kill.email=TRUE
#wrapper.event.jvm_killed.email=TRUE
#wrapper.event.jvm_unexpected_exit.email=TRUE
#wrapper.event.wrapper_stop.email=TRUE# Specify custom mail content
wrapper.event.jvm_restart.email.body=The JVM was restarted.\n\nPlease check on its status.\n#********************************************************************
# Wrapper Windows NT/2000/XP Service Properties
#********************************************************************
# WARNING - Do not modify any of these properties when an application
#  using this configuration file has been installed as a service.
#  Please uninstall the service before modifying this section.  The
#  service can then be reinstalled.# Name of the service
wrapper.name=SparkSubmitService# Display name of the service
wrapper.displayname=Task_SparkSubmitService# Description of the service
wrapper.description=Submit Spark Job# Service dependencies.  Add dependencies as needed starting from 1
wrapper.ntservice.dependency.1=# Mode in which the service is installed.  AUTO_START, DELAY_START or DEMAND_START
wrapper.ntservice.starttype=AUTO_START# Allow the service to interact with the desktop.
wrapper.ntservice.interactive=false

备注:

其中放大加粗红色的配置项是需要需要为自己项目的配置向,其他项可以不修改。

启动服务器:

1、启动前需要先修改权限为777

chmod 777 SparkSubmitService

2、Linux服务启动命令:

[tt@vh01 bin]$ ./SparkSubmitService start
Starting Task_SparkSubmitService...
Waiting for Task_SparkSubmitService.....
running: PID:16531

3、Linux服务停止命令:

[tt@vh01 bin]$ ./SparkSubmitService stop
Stopping Task_SparkSubmitService...
Stopped Task_SparkSubmitService.

Linux服务命令help列表:

[tt@vh01 bin]$ ./SparkSubmitService
Usage: ./SparkSubmitService [ console | start | stop | restart | condrestart | status | install | installstart | remove | dump ]Commands:console      Launch in the current console.start        Start in the background as a daemon process.stop         Stop if running as a daemon or in another console.restart      Stop if running and then start.condrestart  Restart only if already running.status       Query the current status.install      Install to start automatically when system boots.installstart Install and start running as a daemon process.remove       Uninstall.dump         Request a Java thread dump if running.

4、Linux服务状态查看命令:

[tt@vh01 bin]$ ./SparkSubmitService status
Task_SparkSubmitService is not running.

5、CentOS下设置SparkSubmitService为开启启动项:

需要在wrapper的bin目录下执行以下命令:

cp SparkSubmitService  /etc/init.d/SparkSubmitService
cd /etc/rc.d/init.d
chmod +x SparkSubmitService
/sbin/chkconfig --del SparkSubmitService
/sbin/chkconfig --add SparkSubmitService

转载于:https://www.cnblogs.com/yy3b2007com/p/6757117.html

wrapper x64 版本发布到centos相关推荐

  1. ubuntu18.04.4 安装 qq x64 版本

    文章目录 ubuntu18.04.4 安装 qq x64 版本 确定版本 根据你所使用的linux发行版选择格式: 1 代码安装 1 下载 2 安装qq 3 删除qq 2 图形化界面安装 1 下载 2 ...

  2. 禅道 11.4.1 版本发布,主要优化细节

    禅道项目管理软件集产品管理.项目管理.质量管理.文档管理.组织管理和事务管理于一体,是一款功能完备的项目管理软件,完美地覆盖了项目管理的核心流程.禅道官网:www.zentao.net. 大家好,禅道 ...

  3. 禅道项目管理_禅道 11.6.1 版本发布,完善细节,修复 Bug

    禅道项目管理软件集产品管理.项目管理.质量管理.文档管理.组织管理和事务管理于一体,是一款功能完备的项目管理软件,完美地覆盖了项目管理的核心流程.禅道官网:www.zentao.net. 大家好,禅道 ...

  4. Asp.NetCore程序发布到CentOs(含安装部署netcore)--最佳实践(二)

    原文:Asp.NetCore程序发布到CentOs(含安装部署netcore)--最佳实践(二) Asp.NetCore程序发布到CentOs(含安装部署netcore)--最佳实践(一) 接上一篇 ...

  5. bootstrap 新闻列表_kuapingUI 2.2 版本发布,跨屏 UI-bootstrap 大组件 UI 框架

    kuapingUI 2.2 版本发布,增加了一个比较实用的工具栏web组件,是由分享按钮组 + 联系按钮组构成,分享按钮组包含了 QQ空间.朋友圈.QQ.微信.微博等分享:联系按钮组包含了 微信.QQ ...

  6. 多节点CDN缓存加速系统wdcdn2.4版本发布(20111213)

    多节点CDN缓存加速系统wdcdn2.4版本发布(20111213) Wdcdn是一套基于Linux+apache+squid架构开发的CDN缓存加速系统及管理系统 可帮助大中小站长或大中小企业低成本 ...

  7. 使用自动化插件release来管理maven项目的版本发布

    在实际开发过程中,特别是现在推崇的敏捷开发,项目版本的发布非常频繁,有可能是一周就一个版本.对于多模块构建的项目,如何在发布时统一升级项目版本,如果都由人来操作,很容易就漏改某个模块的版本,导致出现问 ...

  8. weblogic创建域后启动不了_WebLogic的Azure虚拟机主要版本发布

    作者:Reza Rahman 我们很高兴地宣布,将发布在Azure Linux虚拟机上运行Oracle WebLogic Server(WLS)的解决方案的主要版本.该版本是与WebLogic团队共同 ...

  9. Istio 1.1 版本发布,性能和可用性提升

    3月20日,Istio 1.1版本发布,距离Istio 1.0版本发布已经过去了7个月.Istio 1.0版本发布的时候,一些主要新功能包括: 将多个 Kubernetes 集群添加到单个网格中,并启 ...

  10. 版本发布后软件测试人员要做的工作

    我在这里讨论的方法主要是针对互联网企业的测试,可能对传统企业的测试来说会有点不同,但是大体上是适用所有公司的测试情况的. 版本发布后大部分测试人员的意识里面都会认为该要好好休息一下了,放几天羊,做做其 ...

最新文章

  1. Csharp: FreeTextbox 编辑器控件运行时错误: 'FTB_ResizeGalleryArea' 未定义
  2. Image Lab 6 for MacOS WIN 图像分析软件下载
  3. 全球首届“AI球球大作战:Go-Bigger多智能体决策智能挑战赛”开启
  4. Send data format set as XML
  5. 【C# 调用 Go 语言】0x2 参数、返回值与类型转换
  6. 易宝典文章——玩转Office 365中的Exchange Online服务 之二十六 根据文本内容筛选群发邮件...
  7. 卸载VMware Server后,无法加载登录用户界面 #F#
  8. 赞!图像生成PyTorch库火了,涵盖18+ SOTA GAN实现
  9. ext2 无损升级 ext3、ext4
  10. VC890D万用表弹片图片
  11. Mysql 导出表结构或数据
  12. 九种机器学习模型的简单介绍
  13. pvs显示unknown device
  14. 三菱FX5U添加新模块
  15. kindle电子书资源搜索及制作教程
  16. Havok物理引擎_百度百科
  17. Java导出excel那些事
  18. android studio 内部存储(将数据储存到文件中)
  19. php与python进行数据交互
  20. 再见了 Docker!K8S 已成气候!

热门文章

  1. asp打开exe执行本地程序._ASP.NET Core Web实现Windows 服务功能
  2. cognos报表导出excel_17个新增功能点,让报表更惊艳!
  3. 对数几率回归(Logistic Regression)
  4. 购物搜索引擎架构的变与不变
  5. 阿里巴巴Java开发 之 MySQL规约
  6. ssh访问限制 /etc/hosts.allow 和/etc/hosts.deny 详解
  7. A New Start
  8. 042 第一章 简介
  9. 【学习0605】NVIDIA DRIVE AGX Developer Kit - How to set up
  10. java.lang.NoSuchFieldError: INSTANCE at org.apache.http.conn.ssl.SSLConnectionSocketFactory