jenkins+kubernetes（k8s）发布Springboot项目

jenkins+kubernetes（k8s）发布Java项目

准备工作
机器列表
镜像列表
创建一个java项目
创建jenkins镜像
Jenkins服务配置
启动jenkins
访问jenkins
系统配置和权限配置
发布Java服务
查看java服务运行状态

记录一下学习k8s+jenkins发布项目的过程，期间遇到很多坑。希望能让刚接触的童鞋少走弯路。

准备工作

本篇只介绍发布的过程，在这之前有很多必要的工作要做：

k8s集群搭建 ，我自己搭的是3个节点的集群；
harbor搭建 ，docker镜像私仓；

机器列表

host	ip	描述
centos7docker	192.168.147.129	harbor私仓机器
k8smaster	192.168.147.130	k8smaster节点
k8sworker1	192.168.147.131	k8s节点
k8sworker2	192.168.147.132	k8s节点

镜像列表

镜像名称	依赖的镜像	描述
jenkins/jenkins:lts	无	jenkins官方镜像
centos7docker:443/aliang-xyl/jenkins:lts	jenkins/jenkins:lts	自制jenkins镜像
centos:7	无	centos7官方镜像
centos7docker:443/aliang-xyl/java8:jdk-8u231	centos:7	自制java服务基础镜像
centos7docker:443/aliang-xyl/provider-docker-demo:e161850	centos7docker:443/aliang-xyl/java8:jdk-8u231	java服务镜像

创建一个java项目

创建一个Java项目，结构如下:

项目Dockerfile文件内容, 用于项目打包成镜像

# 基于哪个镜像, 这个是我自定义的镜像，可根据实际情况更改
FROM centos7docker:443/aliang-xyl/java8:jdk-8u231ARG JAR_FILE
ADD target/provider-docker-demo.jar /provider-docker-demo.jar# 配置容器启动后执行的命令
ENTRYPOINT  ["java","-Ddemo.test=test", "-Xmn256M", "-Xms1G", "-Xmx1G", "-XX:MetaspaceSize=512M","-jar","/provider-docker-demo.jar"]

k8s.ymal文件内容

apiVersion: v1
kind: Service
metadata:name: provider-docker-demonamespace: defaultlabels:app: provider-docker-demo
spec:type: NodePortports:- port: 30001nodePort: 30001 #service对外开放端口selector:app: provider-docker-demo
---
apiVersion: apps/v1
kind: Deployment #对象类型
metadata:name: provider-docker-demo #名称labels:app: provider-docker-demo #标注
spec:replicas: 1 #运行容器的副本数，修改这里可以快速修改分布式节点数量selector:matchLabels:app: provider-docker-demotemplate:metadata:labels:app: provider-docker-demospec:containers: #docker容器的配置- name: provider-docker-demoimage: centos7docker:443/aliang-xyl/provider-docker-demo:<BUILD_TAG> # pull镜像的地址 ip:prot/dir/images:tagimagePullPolicy: IfNotPresent #pull镜像时机，ports:- containerPort: 30001 #容器对外开放端口volumeMounts:- mountPath: logs/privider-demoname: provider-docker-demo-logsvolumes:- name: provider-docker-demo-logshostPath: path: /home/logs/privider-docker-demo

自定义基础docker镜像Dockerfile，定制一个Java服务运行的基础镜像

# centos:7是官方镜像
FROM centos:7
# jdk信息
ADD jdk-8u231-linux-x64.tar.gz /usr/local
ENV JAVA_HOME  /usr/local/jdk1.8.0_231
ENV CLASSPATH $JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
ENV PATH $PATH:$JAVA_HOME/bin
# 更改系统语言，不更改的话java服务中中文会出现乱码
RUN yum -y install kde-l10n-Chinese telnet && \
yum -y install glibc-common &&\
yum clean all  && \
localedef -c -f UTF-8 -i zh_CN zh_CN.utf8
ENV LC_ALL=zh_CN.utf8
ENV LANG=zh_CN.utf8
ENV LANGUAGE=zh_CN.utf8
# 更改时区
RUN echo 'Asia/Shanghai' >/etc/timezone

创建基础镜像

[root@centos7docker home]# docker build -t centos7docker:443/aliang-xyl/java8:jdk-8u231 . -f DockerFile

推送至harbor私仓
这里需要登录至私仓，否则推不上去

[root@centos7docker home]# docker login -uadmin -p123456 centos7docker:443
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded[root@centos7docker home]# docker push centos7docker:443/aliang-xyl/java8:jdk-8u231

基础镜像构建完毕

创建jenkins镜像

官方的镜像可以直接使用,目前我使用的Jenkins版本是2.204.5，但是如果要发布k8s服务的话，会报错：

kubectl command not found

因为没有安装kubectl,所以需要定制jenkins镜像，Dockerfile如下:

# 这个镜像是官方镜像
FROM jenkins/jenkins:lts# 因为默认不是使用root账号，无法完成下面的操作，所以必须使用root
USER root# 直接删除自带的jdk，不同的jenkins可自行运行docker run -it [imageid] /bin/sh  然后查看jdk版本和路径，然后删除
RUN rm -rf /usr/local/openjdk-8# 使用自己的jdk
ADD jdk-8u231-linux-x64.tar.gz /usr/local
ENV JAVA_HOME  /usr/local/jdk1.8.0_231
ENV CLASSPATH $JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
ENV PATH $PATH:$JAVA_HOME/bin# 使用自己的maven,注意先把压缩包里的settings.xml改好,比如本地仓库路径和私服配置信息
# 这里我把仓库路径配置为：/var/jenkins_home/repository
ADD apache-maven-3.6.3-bin.tar.gz /usr/local
ENV MAVEN_HOME /usr/local/apache-maven-3.6.3
ENV PATH $MAVEN_HOME/bin:$PATH# 安装kubeclt工具
ADD kubernetes-client-linux-amd64.tar.gz /
RUN chmod +x ./kubernetes/client/bin/kubectl
RUN mv ./kubernetes/client/bin/kubectl /usr/local/bin/kubectl
RUN ln -s /usr/local/bin/kubectl /usr/bin/kubectl# 需要找到k8s集群里master节点的/etc/kubernetes/admin.conf配置拷贝出来
ADD admin.conf /root
RUN mkdir -p /root/.kube
RUN mv /root/admin.conf /root/.kube/config
RUN chown $(id -u):$(id -g) /root/.kube/config# 更改jenkins镜像的时区
RUN echo 'Asia/Shanghai' >/etc/timezone

准备文件

[root@k8smaster jenkins]# ll
总用量 211704
-rw-r--r-- 1 root root      5449 3月   6 16:16 admin.conf
-rw-r--r-- 1 root root   9506321 3月  21 02:58 apache-maven-3.6.3-bin.tar.gz
-rw-r--r-- 1 root root       726 3月  21 03:29 Dockerfile
-rw-r--r-- 1 root root 194151339 3月   3 16:04 jdk-8u231-linux-x64.tar.gz
-rw-r--r-- 1 root root  13108955 3月  21 02:38 kubernetes-client-linux-amd64.tar.gz

kubectl下载地址:
注意：我使用的是v1.17.3版本，可根据自己的实际版本下载
https://storage.googleapis.com/kubernetes-release/release/v1.17.3/kubernetes-client-linux-amd64.tar.gz

接下来开始构建jenkins镜像
镜像名称:centos7docker:443/aliang-xyl/jenkins:lts

[root@centos7docker home]# docker build -t centos7docker:443/aliang-xyl/jenkins:lts . -f DockerFile
[root@centos7docker home]# docker push centos7docker:443/aliang-xyl/jenkins:lts

Jenkins服务配置

k8s账号角色信息，文件名：jenkins-service-account.yml

apiVersion: v1
kind: ServiceAccount
metadata:name: jenkins-admin
secrets:
- name: jenkins-admin-token-bcv7t---
kind: Role
#rbac的版本为官方定义的版本，可去官网查询
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: jenkins-admin
rules:
- apiGroups: [""]resources: ["pods"]verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]resources: ["pods/exec"]verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]resources: ["pods/log"]verbs: ["get","list","watch"]
- apiGroups: [""]resources: ["secrets"]verbs: ["get"]
- apiGroups: ["","extensions", "apps"]resources: ["deployments"]verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]resources: ["", "services", "endpoints"]verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]  ---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:name: jenkins-admin
roleRef: kind: Rolename: jenkins-adminapiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccountname: jenkins-admin

Jenkins的Deployment和Service，文件:jenkins.ymal
注意：我直接使用的是root权限启动的jenkins(yml里有备注)，可以省去很多权限不足的麻烦

#-----Deployment----------------
apiVersion: apps/v1
kind: Deployment
metadata:name: jenkinslabels: app: jenkins
spec:replicas: 1                #副本数为1selector:matchLabels:app: jenkinstemplate:metadata:labels:app: jenkinsspec:serviceAccountName: jenkins-adminimagePullSecrets:- name: jenkins-admin-token-bcv7tcontainers:- name: jenkinsimage: centos7docker:443/aliang-xyl/jenkins:ltsimagePullPolicy: IfNotPresentsecurityContext:runAsUser: 0   #直接使用root权限启动，可以减少很多权限带来的问题ports:- containerPort: 8080volumeMounts:- mountPath: /var/jenkins_homename: jenkins-home - name: docker   #docker的挂载不能少，否则在运行脚本时会找不到docker命令mountPath: /usr/bin/docker- name: docker-sockmountPath: /var/run/docker.sock- name: date-configmountPath: /etc/localtime- name: kube-configmountPath: /root/.kubehostAliases:          # 在spec下的hostAliases 处添加，此处为虚拟机的host配置，可根据实际情况来配置- ip: "192.168.147.130"hostnames:- "k8s.apiserver"- ip: "192.168.147.131"hostnames:- "k8sworker1"- ip: "192.168.147.132"hostnames:- "k8sworker2"volumes:- name: jenkins-homehostPath: path: /home/data/jenkins_home- name: dockerhostPath:path: /usr/bin/docker- name: docker-sockhostPath:path: /var/run/docker.sock- name: date-confighostPath:path: /etc/localtime- name: kube-confighostPath:path: /root/.kube
---#------service---------------
apiVersion: v1
kind: Service
metadata:name: jenkinslabels:name: jenkins
spec:type: NodePortports:- name: jenkinsport: 8080 targetPort: 8080nodePort: 30009         #开启nodeport- name: jenkins-agentport: 50000 targetPort: 50000nodePort: 30010selector:app: jenkins

启动jenkins

在k8s集群的master节点机器上运行以下命令来发布jenkins服务

[root@k8smaster k8s]# kubectl apply -f jenkins-service-account.yml
[root@k8smaster k8s]# kubectl apply -f jenkins.yaml

查看运行状态

[root@k8smaster k8s]# kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
jenkins-75cb68d55f-7qkbz                1/1     Running   0          4h26m

查看日志获取初始化密码

[root@k8smaster k8s]# kubectl logs jenkins-75cb68d55f-7qkbz

初始化的时候日志里会有一个uuid的字符串，该字符串为初始密码，用来登录jenkins

访问jenkins

可以访问k8s的任意一个子节点来访问jenkins:
jenkins初始化，根据上一步拿到的初始密码来登录jenkins，并获取推荐的插件，
注意：版本过低的jenkins无法安装部分插件，目前我所使用的jenkins为最新版本

系统配置和权限配置

系统设置添加kubernetes云
进入系统管理/系统配置

如果这里连接失败，绝大部分原因是权限的问题，可能是k8s发布jenkins-service-account.yml的时候除了问题

添加git权限
进入凭据

添加git权限

添加harbor账号信息

注意：权限里的id定义之后在后面会用到

发布Java服务

新建一个流水线任务

这里我直接使用的是脚本

// harbor镜像仓库地址信息
def registry = "centos7docker:443"
// 项目
def project = "aliang-xyl"
def app_name = "provider-docker-demo"node(){  stage('1.拉取Git代码'){checkout([$class: 'GitSCM', branches: [[name: '*/master'], [name: '*/develop']], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CleanBeforeCheckout']], submoduleCfg: [], userRemoteConfigs: [[credentialsId: '2e599a5b-f64d-4ef6-992b-f2bff64bb7a9', url: 'git@github.com:aLiang-xyl/provider-docker-demo.git']]])script {build_tag = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim()}image_name = "${registry}/${project}/${app_name}:${build_tag}"}stage('2.java代码编译'){echo "------maven版本、JDK版本-------"sh "mvn --version"sh "mvn clean package -Dmaven.test.skip=true"}stage('3.镜像打包docker build') {echo "镜像: ${image_name}"sh "docker --version"sh "docker build -t ${image_name} . -f src/main/docker/Dockerfile"}stage('4.推镜像仓库Push') {withCredentials([usernamePassword(credentialsId: 'jenkins-harbor-creds', passwordVariable: 'dockerHubPassword', usernameVariable: 'dockerHubUser')]) {sh "docker login -u ${dockerHubUser} -p ${dockerHubPassword} ${registry}"sh "docker push ${image_name}"}}stage('5.部署到k8s') {sh "sed -i 's/<BUILD_TAG>/${build_tag}/' src/main/docker/k8s.yaml"sh "kubectl apply -f src/main/docker/k8s.yaml"echo "项目部署成功"}
}

或者也可以把脚本文件Jenkinsfile放在项目的根目录

注意：脚本里的checkout可以自动生成:

接下来就是见证奇迹的时刻

查看java服务运行状态

[root@k8smaster k8s]# clear
[root@k8smaster k8s]# kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
jenkins-75cb68d55f-7qkbz                1/1     Running   0          5h13m
provider-docker-demo-7f66999f78-jphn7   1/1     Running   0          144m

至此整个发布流程结束