title: Jenkins小坑之执行Shell tags:

• Jenkins
• ExecuteBash
• daemonize
• descriptors categories: ci date: 2017-12-04 22:44:37

背景

1. 之前使用jenkins的大概率是作为远程部署，通常使用over ssh plugin作部署

问题

我们正常使用ssh时当Java应用运行时没有问题 但是在Execute Bash执行时发现输出如下

    Starting ...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    waitting for f6-insurance start...    2017-12-01 17:05:25.031  INFO 10067 --- [           main] b.c.e.u.UndertowEmbeddedServletContainer : Undertow started on port(s) 8078 (http)    f6-insurance start success!

但是当访问该进程是发现并没有该Java进程【初步怀疑脚本出现问题】

分析

1. 手动通过执行脚本发现启动正常
2. 通过jenkins调用发现输出正常但是当jenkins结束后java进程消失
3. 首先怀疑脚本中没有使用后台命令启动【以前ssh碰到】可以参考  解决Linux关闭终端(关闭SSH等)后运行的程序自动停止
4. 但是通过检查脚本使用了nohup 没有问题
5. 多次启动jenkins任务发现都是在任务结束时java进程消失【排除oomkiller&nohup】
6. 经过‘细心’观察发现如下日志

        Process leaked file descriptors. See https://jenkins.io/redirect/troubleshooting/process-leaked-file-descriptors for more information

7. 经过相关文档说明总结如下

        To reliably kill processes spawned by a job during a build, Jenkins contains a bit of native code to list up such processes and kill them.        The reason this problem happens is because of file descriptor leak and how they are inherited from one process to another. Jenkins and the child process are connected by three pipes (stdin/stdout/stderr.) This allows Jenkins to capture the output from the child process. Since the child process may write a lot of data to the pipe and quit immediately after that, Jenkins needs to make sure that it drained the pipes before it considers the build to be over. Jenkins does this by waiting for EOF.        When a process terminates for whatever reasons, the operating system closes all the file descriptors it owned. So even if the process didn't close stdout/stderr, Jenkins will nevertheless get EOF.        The complication happens when those file descriptors are inherited to other processes. Let's say the child process forks another process to the background. The background process (AKA daemon) inherits all the file descriptors of the parent, including the writing side of the stdout/stderr pipes that connect the child process and Jenkins. If the daemon forgets to close them, Jenkins won't get EOF for pipes even when the child process exits, because daemon still have those descriptors open. That's how this problem happens.

> 从官方的说明中可以看出，造成这种问题的原因，是由于文件描述符的丢失以及子进程是如何继承的。
>
> Jenkins和子进程之间的通信使用三根管道进行：stdin，stdout，stderr，由于Jenkins可以捕捉到子进程的输出，而子进程可能往stdout中写入大量数据然后立即退出，Jenkins为了完整的读取子进程的输出，会在用于子进程stdout的管道上等待EOF。这样，无论子进程由于什么原因退出，系统将关闭进程使用的文件描述符，因而Jenkins总是可以收到EOF。
>
> 但是当子进程在后台fork出另一个进程的时候（比如A fork出了B，启动一个daemon进程），情况就出现一些变化。由于进程B会继承进程A的文件描述 符，如果进程B没有关闭这些描述符，即使进程A退出，这些描述符依然是打开的，Jenkins将不会在相应管道上收到EOF。
>
> 通常一个实现良好的daemon会关闭所有文件描述符以避免这个问题，但是总有一些实现没有遵循这个规则。在更旧版本的Hudson上，这个问题甚至将导致Job无法结束，Jenkins一直在那等待EOF。

8. 我们找到一个简单的工具称之为daemonize
9. 安装daemonize

        $ git clone http://github.com/bmc/daemonize.git$ ./configure$ make$ make install

10. 执行脚本时修改为如下

        daemonize -E BUILD_ID=dontKillcenter  /mnt/shell/restart-f6-insurance-web-prod-trial.sh

方案

    #!/bin/bashcp f6-insurance-web/target/f6-insurance-web-0.0.1-SNAPSHOT.jar /mnt/f6_jar/f6-insurance-web-0.0.1-SNAPSHOT-prod-trial.jarcd /mnt/f6-insurance-8078-prod-trial/daemonize -E BUILD_ID=dontKillcenter  /mnt/shell/restart-f6-insurance-web-prod-trial.sh

这样就可以成功运行！