Jenkins Pipeline & Kubernetes 如何创建 pod

文章目录

Jenkins Pipeline & Kubernetes 如何创建 pod
- 1. 前言
- 2. Jenkins 插件
- - 2.1 安装 Kubernets Plugin
  - 2.2 安装 Docker Plugin
  - 2.3 安装 Git Plugin
- 3. Jenkins 配置 kubernetes credentials
- 4. Jenkins 连接 minikube 集群
- 5. 配置参数说明
- - 5.1 Pod template 参数
  - 5.2 Container template 参数
- 6. Jenkins Piepline 部署 pod 实例
- - 6.1 创建一个简单 pod
  - 6.2 pod name 变化
  - 6.3 指定 namespace
  - 6.4 volumes 挂载
  - 6.5 Liveness Probe 探针
  - 6.6 创建 pod 内多容器
  - 6.7 继承
  - 6.8 pod 嵌套
  - 6.9 Pipeline script from SCM
  - 6.10 配置共享库
  - 6.11 声明式 pipeline
- 7 Jenkins Pipeline 构建镜像实例
- - 7.1 git 拉取仓库 & 构建镜像
  - 7.2 编写 Dockerfile & 构建镜像
  - 7.3 git 拉取仓库 & kaniko 构建镜像 & 推送入库
- 问题
- - http://192.168.10.90:8080/tcpSlaveAgentListener/ is invalid: 404 Not Found

关于kubernetes & Jenkins 部署可以参考：minikube & helm 安装 jenkins

1. 前言

在 DevOps 的世界里，自动化是主要目标之一。针对 CI/CD 的最著名的开源工具之一就是自动化服务器 Jenkins。从简单的 CI 服务器到完整的 CD 集线器，Jenkins 都可以处理。

CI：持续集成（continuous integration）是在源代码变更后自动检测、拉取、构建和（在大多数情况下）进行单元测试的过程。目标是快速确保开发人员新提交的变更是好的，并且适合在代码库中进一步使用
CD：持续交付（continuous delivery）通常是指整个流程链（管道），它自动监测源代码变更并通过构建、测试、打包和相关操作运行它们以生成可部署的版本，基本上没有任何人为干预。持续交付在软件开发过程中的目标是自动化、效率、可靠性、可重复性和质量保障（通过持续测试）。
CD: 持续部署（continuous deployment）是指能够自动提供持续交付管道中发布版本给最终用户使用的想法。根据用户的安装方式，可能是在云环境中自动部署、app 升级（如手机上的应用程序）、更新网站或只更新可用版本列表。
Pipeline: 将源代码转换为可发布产品的多个不同的任务(task)和作业(job)通常串联成一个软件“管道”，一个自动流程成功完成后会启动管道中的下一个流程。这些管道有许多不同的叫法，例如持续交付管道、部署管道和软件开发管道。

“下面我们将实践如何利用 Jenkins 强大的 CI/CD 特性来练习如何部署 kubernetes 应用作为开始。”

2. Jenkins 插件

2.1 安装 Kubernets Plugin

2.2 安装 Docker Plugin

2.3 安装 Git Plugin

3. Jenkins 配置 kubernetes credentials

获取 token

 kubectl get secrets -n jenkins jenkins-token-6r26g -oyaml
apiVersion: v1
data:ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCakNDQWU2Z0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwdGFXNXAKYTNWaVpVTkJNQjRYRFRJeU1URXlPVEEyTURJd09Gb1hEVE15TVRFeU56QTJNREl3T0Zvd0ZURVRNQkVHQTFVRQpBeE1LYldsdWFXdDFZbVZEUVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTU8xCkpIMER5VU5DeGh2dHdtR05EaGxORTZFTXJWMzl0N1dmUjJmMTJEM3U4SVlpckFhdFBNb1RYZThpTDR4NXl5ckoKaUhRdWtIMXAzcjhqS0E4WVpaa2cwN3FIOE1mYXB2dG9qQjQ3QUROanBLYUNVcXh6UFlvY3l0VlU3UDA0dDhMVQpyRTZFTG9qcGlWcWNEdzZSakhEQ3p2R3NFU1NvTUIyZTlxVXN2dU9kMmdlMFVwMnJKdjRmTTY3aEdJN3FIYkJ3CjVFbE5tMzcySytxOS9nUFBtSW1kQU94Y2xFOENTcy9aYWxuV1AzcWdLbGVoNnQxbkFscnhpbXpaSVdkYnZvMzYKemxTRmwwaHlrZGNWL0RUdndaNFYvMkR1OVVtbWpsWFJ5cWdOYXF1R2lTYXVMRjhWRkpYWnROWmhSR2NWZW1tNQpRQ0gyWWFEN3lzMmJOYlM3YUtzQ0F3RUFBYU5oTUY4d0RnWURWUjBQQVFIL0JBUURBZ0trTUIwR0ExVWRKUVFXCk1CUUdDQ3NHQVFVRkJ3TUNCZ2dyQmdFRkJRY0RBVEFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVcKQkJSVXg5bTFqVlIrR1llNXJMalZhRmpxaXljbjZEQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUF3M3JpRkY1QgozTXZSZmFlRnB0d3pRRzYzVEZncVoxN3luaDBSV2xrUkM3M3c0L1BqczhUUmhXRFNYTy94elNROTNJNjNlaHkwCldIOTJsMDJhN3NNTXVvY1YrRG8vTUlpVmI3bVA2VmtKbjF2Vm43cE4zM2c5dDJFWkY2Yis0Q0JUUUo5YXlodGwKMHo5Y3hwMXVaMEt4SUZ1bzJuY3lPREFEZ293T3h0WHJKZ1h5ckE5MldFbi91NUl5VnJ1Q2ZlWjk0YTMwN3NVUAplMXpnTlBBam8xUHdTQVY4bE5tRkpZbG9FVzNhRkE1MDU3STZYZy81dEg0QzJ2c3NkNVFxS2R4RHM1UU5QK2VpClhZblp2S2dTTmJSVUtyL1NIM1Q5NkQ0azIrQXpsWEhBYVRwYk9uQURzYmt3YTYwcHUvWmJsa0dLajhFVE9KL2UKRldleGYyUmtzbkVWN1E9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==namespace: amVua2lucw==token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNkltdHpSM2xwUTJrd2FUUmxiR3gyYkMwMFIyVndOR0ZUZEZsVlVXUTRVaTFNUTBnMU9ISlBiRkJOWldNaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpxWlc1cmFXNXpJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbXBsYm10cGJuTXRkRzlyWlc0dE5uSXlObWNpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pYW1WdWEybHVjeUlzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG5WcFpDSTZJak01TnpjMU0yUmtMVEJtTTJZdE5ERmxNaTFoTnpOakxUaGhOVEV6TldRMlptRmhZU0lzSW5OMVlpSTZJbk41YzNSbGJUcHpaWEoyYVdObFlXTmpiM1Z1ZERwcVpXNXJhVzV6T21wbGJtdHBibk1pZlEuamx6SjV1bmRlZ3ZUMzVmVmViY3JsWnUxa09talAwTGpLenhsb2sxTHhINVowcm9jTmVsSmVJMXM4NWtnT1JXZGNuanFaTlg3VHpxano1eml1OS1peGZuY1UtTVRicTlRcTVMcnMyd1c1cGRTZTFwSlpWcmx5X2pGN2tOdXFSWENZZkdWVUFwbGFELXNES2ZJdFhmdEJ2dHpvWTgzV0QxdEdZMkJkSzRKM3NGdDRIdmw2YUVuUlZFejJ5WlRBTVBKZEJ1NUI4NjJCUnFOMEFicG13UFByM3FtellaZGZVQzNzdlBIVE9BWGxxUUpUcHRQbDVmMS05dEtKZUlGUGNLY1F5WTZSc0RXbUFfRW1sTHV6MWJWRkpxV2pxWnVqa3cyTndYeXhvS1VMNjBhVm9Lc2dQTVFpTU44TEg4Z2s0bTg5STQ5VjVvb2NpX3N3VTNYVy05cjNR
kind: Secret
metadata:annotations:kubernetes.io/service-account.name: jenkinskubernetes.io/service-account.uid: 397753dd-0f3f-41e2-a73c-8a5135d6faaacreationTimestamp: "2022-12-03T13:49:22Z"name: jenkins-token-6r26gnamespace: jenkinsresourceVersion: "199550"uid: 0b896aba-1810-436a-9eb8-dd0502e9e79f
type: kubernetes.io/service-account-token

复制token内容至secret

具体可官方添加 credentials 步骤

4. Jenkins 连接 minikube 集群

我们需要配置 manage Nodes and Clouds

name: 定义集群名称即可。
kubernetes credentials 选择下拉刚刚创建的 minikube2
Kubernetes URL 与通过命令 kubectl config view获取
填完点击测试一下，再保存。
确保 websocket 勾选，否则，pod内无法运行jnlp代理。

$ kubectl  config view
apiVersion: v1
clusters:
- cluster:certificate-authority: /root/.minikube/ca.crtextensions:- extension:last-update: Wed, 30 Nov 2022 14:20:59 CSTprovider: minikube.sigs.k8s.ioversion: v1.28.0name: cluster_infoserver: https://192.168.10.26:8443name: minikube
contexts:
- context:cluster: minikubeextensions:- extension:last-update: Wed, 30 Nov 2022 14:20:59 CSTprovider: minikube.sigs.k8s.ioversion: v1.28.0name: context_infonamespace: defaultuser: minikubename: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikubeuser:client-certificate: /root/.minikube/profiles/minikube/client.crtclient-key: /root/.minikube/profiles/minikube/client.key

5. 配置参数说明

5.1 Pod template 参数

cloud ：Jenkins设置中定义的云的名称。默认为kubernetes
name： pod名称.
namespace ：pod 命名空间.
label ：节点标签。这就是在通过节点步骤请求代理时可以引用 pod 模板的方式。在管道中，建议省略此字段并依赖生成的标签，该标签可以使用 podTemplate 块中定义的 POD_LABEL 变量引用。
yaml Pod 的 yaml 表示，以允许设置任何不支持的值作为字段
yamlMergeStrategy ：merge() or override(). 控制 yaml 定义是否覆盖或与从使用 inheritFrom 声明的 pod 模板继承的 yaml 定义合并。默认为 override() （出于向后兼容性原因）。
containers： container templates 部分
serviceAccount ：pod 服务帐户.
nodeSelector ：pod的节点选择器.
nodeUsageMode：要么NORMAL要么EXCLUSIVE，该参数控制的是只调度标签表达式匹配的作业还是尽可能使用节点.
volumes：挂载持久存储卷
- configMapVolume : 挂载ConfigMap，只读.
- dynamicPVC() : 动态管理的持久卷 pvc ，它与 pod 同时被删除.
- emptyDirVolume (default): 空目录
- hostPathVolume() : 挂载主机目录
- nfsVolume() : 挂载NFS目录
- persistentVolumeClaim() : 绑定pvc.
- secretVolume : 挂载 secret，只读，适用于证书、用户密码.
envVars：应用于所有容器的环境变量.
- envVar：个环境变量，其值是内联定义的.
- secretEnvVar：通过secret对象定义一个环境变量.
imagePullSecrets：提取 secret 名称, 从私有 Docker 仓库中提取镜像.
annotations：pod注释
inheritFrom：要继承的一个或多个 pod 模板的列表
slaveConnectTimeout ：agent 在线超时秒数
podRetention：控制保留代理 pod 的行为。可以是 ‘never()’、'onFailure()'、‘always()’ 或 ‘default()’ - 如果为空，将默认在 activeDeadlineSeconds 过后删除 pod。
activeDeadlineSeconds：如果 podRetention 设置为 never() 或 onFailure()，则 pod 将在截止日期过后删除。
idleMinutes：允许 pod 保持活动状态以供重用，直到自上一个步骤执行后配置的分钟数过去。
showRawYaml：启用或禁用原始pod清单（manifest）的输出。默认为true
runAsUser：定义运行pod中所有容器的用户ID.
runAsGroup：定义在pod中运行所有容器的组ID.
hostNetwork：主机网络.
workspaceVolume：定义workspace卷的类型.
- dynamicPVC() ：挂载动态管理的pvc，会与pod一同被删除.
- emptyDirWorkspaceVolume (default): 定义主机上分配的空目录
- hostPathWorkspaceVolume() : 挂载主机目录
- nfsWorkspaceVolume() : 挂载nfs volume
- persistentVolumeClaimWorkspaceVolume() : 绑定pvc.

5.2 Container template 参数

Container template可通过用户界面与pipeline配置。

name：容器名字.
image ：镜像名称.
envVars ：应用于容器的环境变量(补充和覆盖在pod已设置的环境变量).
- envVar：一个环境变量，其值是内联定义的.
- secretEnvVar ：通过secret对象定义一个环境变量…
command：容器将执行的命令。将覆盖 Docker 入口点。常用命令：sleep.
args ：传递给命令的参数。例如：99999999 .
ttyEnabled：标志，以标记tty应该启用.
livenessProbe ：探针（不支持 - httpGet liveness probes）
ports：暴露容器上的端口
alwaysPullImage：容器将在启动时拉取镜像.
runAsUser：定义运行容器的用户ID.
runAsGroup：定义运行容器的组ID.

6. Jenkins Piepline 部署 pod 实例

6.1 创建一个简单 pod

创建一个包含 docker 容器的 pod ，并且带有build标签。创建完后，使用docker version命令执行查看容器内docker 的版本信息。

Pipeline script：

podTemplate(label: 'build', containers: [containerTemplate(name: 'docker', image: 'docker', command: 'cat', ttyEnabled: true)],volumes: [hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),]) {node('build') {container('docker') {sh 'docker version'}        }  }

点击执行“build”

构建成功。

minikube 集群查看部署结果：

$ k get pods -n jenkins
NAME                READY   STATUS    RESTARTS   AGE
build-cb6cf-twpw7   2/2     Running   0          6s

6.2 pod name 变化

创建pod，编写podTemplate，通过yaml 定义 pod内容。

podTemplate(yaml: '''apiVersion: v1kind: Podmetadata:labels: some-label: some-label-valuespec:containers:- name: busyboximage: busyboxcommand:- sleepargs:- 99d''') {node(POD_LABEL) {container('busybox') {echo POD_CONTAINER // displays 'busybox'sh 'hostname'}}
}

第一次构建部署输出：

$ k get pods -n jenkins | grep busybox
busybox-1-zdnpq-mwqtd-0vfg0   2/2     Running   0          19s

busybox pod前缀默认是项目名字
1 代表第一次构建
zdnpq-mwqtd-0vfg0是随机码

当我们第二次构建，pod 名字会发生变化，保持pod名字的唯一性。

$  k get pods -n jenkins | grep busybox
busybox-2-c7blp-840k1-s5c5d   2/2     Running   0          20s

我们尝试修改项目名称为busybox2

输出结果

$ k get pods -n jenkins | grep busybox
busybox2-3-2fc4l-8z8dp-2t0c7   2/2     Running   0          7s

原来的busybox变成busybox2，并且表明第三次构建。

我们尝试添加定义pod的name为busybox3

podTemplate(name: "busybox3",yaml: '''apiVersion: v1kind: Podmetadata:labels: some-label: some-label-valuespec:containers:- name: busyboximage: busyboxcommand:- sleepargs:- 99d''') {node(POD_LABEL) {container('busybox') {echo POD_CONTAINER // displays 'busybox'sh 'hostname'}}
}

当我们自定义pod 的name，构建次数会取消，项目名字不再是pod的前缀。

k get pods -n jenkins | grep busybox
busybox3-lzhfl-7hflx   2/2     Running   0          20s

6.3 指定 namespace

然后，我们尝试换一个已创建好的名字为one的namespace进行构建。

podTemplate(name: "busybox4",namespace: "one",yaml: '''....

$ k get pods -n one  | grep busybox
busybox4-fxd38-5vpwf   2/2     Running   0          24s

6.4 volumes 挂载

挂载本地 /var/run/docker.sock 文件

podTemplate(name: "busybox5",namespace: "one",yaml: '''apiVersion: v1kind: Podmetadata:labels: some-label: some-label-valuespec:containers:- name: busyboximage: busyboxcommand:- sleepargs:- 99d''',volumes: [hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),]) {node(POD_LABEL) {container('busybox') {echo POD_CONTAINER // displays 'busybox'sh 'hostname'}}
}

6.5 Liveness Probe 探针

podTemplate(name: "busybox6",namespace: "one",
containers: [
containerTemplate(name: 'busybox', image: 'busybox', command: 'sleep', args: '99d',livenessProbe: containerLivenessProbe(execArgs: 'ping 127.0.0.1', initialDelaySeconds: 30, timeoutSeconds: 1, failureThreshold: 3, periodSeconds: 10, successThreshold: 1)
)]
){node(POD_LABEL) {container('busybox') {echo POD_CONTAINER // displays 'busybox'sh 'hostname'}}
}

更多kubernetes探针内容：Configure Liveness, Readiness and Startup Probes

6.6 创建 pod 内多容器

podTemplate(name: "build1",namespace: "one",containers: [containerTemplate(name: 'maven', image: 'maven:3.8.1-jdk-8', command: 'sleep', args: '99d'),containerTemplate(name: 'golang', image: 'golang:1.16.5', command: 'sleep', args: '99d')]) {node(POD_LABEL) {stage('Get a Maven project') {git 'https://github.com/jenkinsci/kubernetes-plugin.git'container('maven') {stage('Build a Maven project') {sh 'mvn -B -ntp clean install'}}}stage('Get a Golang project') {git url: 'https://github.com/hashicorp/terraform.git', branch: 'main'container('golang') {stage('Build a Go project') {sh '''mkdir -p /go/src/github.com/hashicorpln -s `pwd` /go/src/github.com/hashicorp/terraformcd /go/src/github.com/hashicorp/terraform && make'''}}}}
}

查看创建的 pod

$ kubectl get pods -n one
NAME                 READY   STATUS    RESTARTS   AGE
build1-qc5pr-679mv   3/3     Running   0          17m
#三个容器
$ kubectl get pods -n one -o jsonpath="{.items[*].spec.containers[*].name}" | tr -s '[[:space:]]' '\n' |sort |uniq
golang
jnlp
maven

注意：jnlp容器是Jenkins代理,并且命令默认将在运行Jenkins代理的jnlp容器中执行。(jnlp的名称是历史的，为了兼容而保留。)，例如：

podTemplate {node(POD_LABEL) {stage('Run shell') {sh 'echo hello world'}}
}

我们还可以通过yaml定义格式：

podTemplate(name: "build1",namespace: "one",yaml: '''apiVersion: v1kind: Podspec:containers:- name: mavenimage: maven:3.8.1-jdk-8command:- sleepargs:- 99d- name: golangimage: golang:1.16.5command:- sleepargs:- 99d
''') {node(POD_LABEL) {stage('Get a Maven project') {git 'https://github.com/jenkinsci/kubernetes-plugin.git'container('maven') {stage('Build a Maven project') {sh 'mvn -B -ntp clean install'}}}stage('Get a Golang project') {git url: 'https://github.com/hashicorp/terraform-provider-google.git', branch: 'main'container('golang') {stage('Build a Go project') {sh '''mkdir -p /go/src/github.com/hashicorpln -s `pwd` /go/src/github.com/hashicorp/terraformcd /go/src/github.com/hashicorp/terraform && make'''}}}}
}

6.7 继承

pod template 可以继承现有模板，也可以不继承。这意味着pod template将从它所继承的模板继承node selector, service account, image pull secrets, container templates and volumes。

yaml根据yamlMergeStrategy的值进行合并。

我通过界面Dashboard > Manage > Jenkins > Configure Clouds创建 pod template

这里我只想将maven的镜像版本由3.8.1-jdk-8升级为3.8.1-jdk-11，在项目编写pipeline script：

podTemplate(name: "build2",namespace: "one",inheritFrom: 'mypod', containers: [containerTemplate(name: 'maven', image: 'maven:3.8.1-jdk-11')]) {node(POD_LABEL) {stage('Get a Maven project') {git 'https://github.com/jenkinsci/kubernetes-plugin.git'container('maven') {stage('Build a Maven project') {sh 'mvn -B -ntp clean install'}}}stage('Get a Golang project') {git url: 'https://github.com/hashicorp/terraform-provider-google.git', branch: 'main'container('golang') {stage('Build a Go project') {sh '''mkdir -p /go/src/github.com/hashicorpln -s `pwd` /go/src/github.com/hashicorp/terraformcd /go/src/github.com/hashicorp/terraform && make'''}}}}
}

$ kubectl get pods -n one
NAME                 READY   STATUS    RESTARTS   AGE
build2-4p074-b1skx   3/3     Running   0          11m$ kubectl get pods build2-4p074-b1skx -n one -o jsonpath="{.spec.containers[*].image}" | tr -s '[[:spa
ce:]]' '\n' |sort |uniq
golang:1.16.5
jenkins/inbound-agent:4.11-1-jdk11
maven:3.8.1-jdk-11

这里看到：通过inheritFrom: 'mypod'实现对模板的继承，并且实现mypod模板中的maven版本v1.8.1-jdk-8升级成为3.8.1-jdk-11，完成覆盖。

6.8 pod 嵌套

通过两个 pod template 方法合成一个包含两个容器的 pod实例。

podTemplate(name: 'docker', namespace: 'one', containers: [containerTemplate(image: 'docker', name: 'docker', command: 'cat', ttyEnabled: true)]) {podTemplate(name: 'maven', namespace: 'one', containers: [containerTemplate(image: 'maven:3.8.1-jdk-11', name: 'maven', command: 'cat', ttyEnabled: true)]) {node(POD_LABEL) {container('docker') {sh "echo hello from $POD_CONTAINER" // displays 'hello from docker'}container('maven') {sh "echo hello from $POD_CONTAINER" // displays 'hello from maven'}}}
}

构建结果：

$ k get pods -n one
NAME                READY   STATUS    RESTARTS   AGE
maven-9h33d-t39gn   3/3     Running   0          5s#显示pod容器名称
$ kubectl get pods maven-9h33d-t39gn -n one -o jsonpath="{.spec.containers[*].name}" | tr -s '[[:space:]]' '\n' | sort |uniq
docker
jnlp
maven

console output 输出：

Started by user Jenkins Admin
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {[Pipeline] podTemplate
[Pipeline] {[Pipeline] node
Created Pod: minikube2 one/maven-1bmzz-wv1sp
Agent maven-1bmzz-wv1sp is provisioned from template maven-1bmzz
---
apiVersion: "v1"
kind: "Pod"
metadata:annotations:buildUrl: "http://jenkins.jenkins.svc.cluster.local:8080/job/busybox2/30/"runUrl: "job/busybox2/30/"labels:jenkins/jenkins-jenkins-agent: "true"jenkins/label-digest: "4ba5a4e248fa7baa06e229917703aa75dbced0ac"jenkins/label: "busybox2_30-lt9wl"name: "maven-1bmzz-wv1sp"namespace: "one"
spec:containers:- command:- "cat"image: "maven:3.8.1-jdk-11"imagePullPolicy: "IfNotPresent"name: "maven"resources:limits: {}requests: {}tty: truevolumeMounts:- mountPath: "/home/jenkins/agent"name: "workspace-volume"readOnly: false- command:- "cat"image: "docker"imagePullPolicy: "IfNotPresent"name: "docker"resources:limits: {}requests: {}tty: truevolumeMounts:- mountPath: "/home/jenkins/agent"name: "workspace-volume"readOnly: false- env:- name: "JENKINS_SECRET"value: "********"- name: "JENKINS_TUNNEL"value: "jenkins-agent.jenkins.svc.cluster.local:50000"- name: "JENKINS_AGENT_NAME"value: "maven-1bmzz-wv1sp"- name: "JENKINS_NAME"value: "maven-1bmzz-wv1sp"- name: "JENKINS_AGENT_WORKDIR"value: "/home/jenkins/agent"- name: "JENKINS_URL"value: "http://jenkins.jenkins.svc.cluster.local:8080/"image: "jenkins/inbound-agent:4.11-1-jdk11"name: "jnlp"resources:limits: {}requests:memory: "256Mi"cpu: "100m"volumeMounts:- mountPath: "/home/jenkins/agent"name: "workspace-volume"readOnly: falsenodeSelector:kubernetes.io/os: "linux"restartPolicy: "Never"volumes:- emptyDir:medium: ""name: "workspace-volume"Running on maven-1bmzz-wv1sp in /home/jenkins/agent/workspace/busybox2
[Pipeline] {[Pipeline] container
[Pipeline] {[Pipeline] sh
+ echo hello from docker
hello from docker
[Pipeline] }
[Pipeline] // container
[Pipeline] container
[Pipeline] {[Pipeline] sh
+ echo hello from maven
hello from maven
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS

6.9 Pipeline script from SCM

github 仓库：https://github.com/jenkinsci/kubernetes-plugin.git

examples/containerLog.groovy

podTemplate(yaml: '''apiVersion: v1kind: Podmetadata:labels:some-label: some-label-valuespec:containers:- name: mavenimage: maven:3.8.1-jdk-8command:- sleepargs:- 99dtty: true- name: mongoimage: mongo
''') {node(POD_LABEL) {stage('Integration Test') {try {container('maven') {sh 'nc -z localhost:27017 && echo "connected to mongo db"'// sh 'mvn -B clean failsafe:integration-test' // real integration testdef mongoLog = containerLog(name: 'mongo', returnLog: true, tailingLines: 5, sinceSeconds: 20, limitBytes: 50000)assert mongoLog.contains('connection accepted from 127.0.0.1:')sh 'echo failing build; false'}} catch (Exception e) {containerLog 'mongo'throw e}}}
}

在console output可以看到输出日志：

[Pipeline] containerLog
.....
> start log of container 'mongo' in pod 'hello-34-p3hfk-8b868-bln0j'
{"t":{"$date":"2022-12-12T04:21:11.334+00:00"},"s":"I",  "c":"NETWORK",  "id":4915701, "ctx":"-","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":13},"outgoing":{"minWireVersion":0,"maxWireVersion":13},"isInternalClient":true}}}
{"t":{"$date":"2022-12-12T04:21:11.336+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2022-12-12T04:21:11.336+00:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
.....
{"t":{"$date":"2022-12-12T04:22:13.070+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":"[1670818933:69974][1:0x7f3fb7217700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 34, snapshot max: 34 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 1"}}
> end log of container 'mongo' in pod 'hello-34-p3hfk-8b868-bln0j'
.......

6.10 配置共享库

示例：利用 Ghostwritten/jenkins-shared-library 作为我的共享库，点击配置Dashboard > Manage Jenkins > Configure System

pod 模板位置：jenkins-shared-library/src/com/kubernetes/PodTemplates.groovy

package com.ghostwritten.PodTemplatespublic void dockerTemplate(body) {podTemplate(containers: [containerTemplate(name: 'docker', image: 'docker:19.03.1', command: 'sleep', args: '99d')],volumes: [hostPathVolume(hostPath: '/var/run/docker.sock', mountPath: '/var/run/docker.sock')]) {body.call()
}
}public void mavenTemplate(body) {podTemplate(containers: [containerTemplate(name: 'maven', image: 'maven:3.8.1-jdk-8', command: 'sleep', args: '99d')]) {body.call()
}
}

注意，POD_LABEL将是生成的最里面的标签，以获得节点上所有外部pod都可用的节点，Jenkins Pipeline script代码：

@Library('my-shared-library') _
import com.ghostwritten.kubernetes.PodTemplatespodTemplates = new PodTemplates()podTemplates.dockerTemplate {podTemplates.mavenTemplate {node(POD_LABEL) {container('docker') {sh "echo hello from $POD_CONTAINER" // displays 'hello from docker'}container('maven') {sh "echo hello from $POD_CONTAINER" // displays 'hello from maven'}}}
}

console output：

Started by user ghostwritten
Loading library my-shared-library@main
Examining Ghostwritten/jenkins-shared-library
Attempting to resolve main as a branch
Resolved main as branch main at revision ecf9353831650f5b7a2a97b316281407fd7de149
The recommended git tool is: NONE
using credential github-token> git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/pod1@libs/41ff7c0a7b996db58dc106402d20945af1169af9339be67d96c4a407546ef059/.git # timeout=10
Fetching changes from the remote Git repository> git config remote.origin.url https://github.com/Ghostwritten/jenkins-shared-library.git # timeout=10
Fetching without tags
Fetching upstream changes from https://github.com/Ghostwritten/jenkins-shared-library.git> git --version # timeout=10> git --version # 'git version 2.38.1'
using GIT_ASKPASS to set credentials github-token> git fetch --no-tags --force --progress -- https://github.com/Ghostwritten/jenkins-shared-library.git +refs/heads/main:refs/remotes/origin/main # timeout=10
Checking out Revision ecf9353831650f5b7a2a97b316281407fd7de149 (main)> git config core.sparsecheckout # timeout=10> git checkout -f ecf9353831650f5b7a2a97b316281407fd7de149 # timeout=10
Commit message: "fix PodTemplate.groovy"> git rev-list --no-walk 3058774daa33ed501231e505eb1c61a20b6404d4 # timeout=10
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {[Pipeline] podTemplate
[Pipeline] {[Pipeline] node
Created Pod: minikube1 default/pod1-10-0x0zw-81s8v-3jw0q
Agent pod1-10-0x0zw-81s8v-3jw0q is provisioned from template pod1_10-0x0zw-81s8v
---
apiVersion: "v1"
kind: "Pod"
metadata:annotations:buildUrl: "http://192.168.10.90:8080/job/pod1/10/"runUrl: "job/pod1/10/"labels:jenkins: "slave"jenkins/label-digest: "71b3bbb3338de8a4c7c7a814e453cd35570e7604"jenkins/label: "pod1_10-0x0zw"name: "pod1-10-0x0zw-81s8v-3jw0q"namespace: "default"
spec:containers:- args:- "99d"command:- "sleep"image: "maven:3.8.1-jdk-8"imagePullPolicy: "IfNotPresent"name: "maven"resources:limits: {}requests: {}tty: falsevolumeMounts:- mountPath: "/var/run/docker.sock"name: "volume-0"readOnly: false- mountPath: "/home/jenkins/agent"name: "workspace-volume"readOnly: false- args:- "99d"command:- "sleep"image: "docker:19.03.1"imagePullPolicy: "IfNotPresent"name: "docker"resources:limits: {}requests: {}tty: falsevolumeMounts:- mountPath: "/var/run/docker.sock"name: "volume-0"readOnly: false- mountPath: "/home/jenkins/agent"name: "workspace-volume"readOnly: false- env:- name: "JENKINS_SECRET"value: "********"- name: "JENKINS_AGENT_NAME"value: "pod1-10-0x0zw-81s8v-3jw0q"- name: "JENKINS_WEB_SOCKET"value: "true"- name: "JENKINS_NAME"value: "pod1-10-0x0zw-81s8v-3jw0q"- name: "JENKINS_AGENT_WORKDIR"value: "/home/jenkins/agent"- name: "JENKINS_URL"value: "http://192.168.10.90:8080/"image: "jenkins/inbound-agent:4.11-1-jdk11"name: "jnlp"resources:limits: {}requests:memory: "256Mi"cpu: "100m"volumeMounts:- mountPath: "/var/run/docker.sock"name: "volume-0"readOnly: false- mountPath: "/home/jenkins/agent"name: "workspace-volume"readOnly: falsenodeSelector:kubernetes.io/os: "linux"restartPolicy: "Never"volumes:- hostPath:path: "/var/run/docker.sock"name: "volume-0"- emptyDir:medium: ""name: "workspace-volume"Running on pod1-10-0x0zw-81s8v-3jw0q in /home/jenkins/agent/workspace/pod1
[Pipeline] {[Pipeline] container
[Pipeline] {[Pipeline] sh
+ echo hello from docker
hello from docker
[Pipeline] }
[Pipeline] // container
[Pipeline] container
[Pipeline] { (hide)
[Pipeline] sh
+ echo hello from maven
hello from maven
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS

有了共享库，我们就可以积累自己常用的pipeline script demo进行封装改为可调用配置。对提高编写Jenkinsfile的能力特别有帮助。

Extending with Shared Libraries

6.11 声明式 pipeline

声明式 pipeline可以很好地利用 kubernets 插件在kubernetes集群完成编译、打包、构建、测试、部署等一些流程。
这里先不展开细写，咱先记录几种 pipeline script 格式。

pipeline {agent {kubernetes {yaml '''apiVersion: v1kind: Podmetadata:labels:some-label: some-label-valuespec:containers:- name: mavenimage: maven:alpinecommand:- cattty: true- name: busyboximage: busyboxcommand:- cattty: true'''}}stages {stage('Run maven') {steps {container('maven') {sh 'mvn -version'}container('busybox') {sh '/bin/busybox'}}}}
}

console output:

Started by user ghostwritten
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {[Pipeline] node
Created Pod: minikube1 default/pod2-8-nlldp-m2ghs-d7ms9
Agent pod2-8-nlldp-m2ghs-d7ms9 is provisioned from template pod2_8-nlldp-m2ghs
---
apiVersion: "v1"
kind: "Pod"
metadata:annotations:buildUrl: "http://192.168.10.90:8080/job/pod2/8/"runUrl: "job/pod2/8/"labels:some-label: "some-label-value"jenkins: "slave"jenkins/label-digest: "90c685e1f5f0ea5bd4929c0bc949c47207784626"jenkins/label: "pod2_8-nlldp"name: "pod2-8-nlldp-m2ghs-d7ms9"namespace: "default"
spec:containers:- command:- "cat"image: "maven:alpine"name: "maven"tty: truevolumeMounts:- mountPath: "/home/jenkins/agent"name: "workspace-volume"readOnly: false- command:- "cat"image: "busybox"name: "busybox"tty: truevolumeMounts:- mountPath: "/home/jenkins/agent"name: "workspace-volume"readOnly: false- env:- name: "JENKINS_SECRET"value: "********"- name: "JENKINS_AGENT_NAME"value: "pod2-8-nlldp-m2ghs-d7ms9"- name: "JENKINS_WEB_SOCKET"value: "true"- name: "JENKINS_NAME"value: "pod2-8-nlldp-m2ghs-d7ms9"- name: "JENKINS_AGENT_WORKDIR"value: "/home/jenkins/agent"- name: "JENKINS_URL"value: "http://192.168.10.90:8080/"image: "jenkins/inbound-agent:4.11-1-jdk11"name: "jnlp"resources:limits: {}requests:memory: "256Mi"cpu: "100m"volumeMounts:- mountPath: "/home/jenkins/agent"name: "workspace-volume"readOnly: falsenodeSelector:kubernetes.io/os: "linux"restartPolicy: "Never"volumes:- emptyDir:medium: ""name: "workspace-volume"Running on pod2-8-nlldp-m2ghs-d7ms9 in /home/jenkins/agent/workspace/pod2
[Pipeline] {[Pipeline] stage
[Pipeline] { (Run maven)
[Pipeline] container
[Pipeline] {[Pipeline] sh
+ mvn -version
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-04T19:00:29Z)
Maven home: /usr/share/maven
Java version: 1.8.0_212, vendor: IcedTea, runtime: /usr/lib/jvm/java-1.8-openjdk/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.0.10-1.el7.elrepo.x86_64", arch: "amd64", family: "unix"
[Pipeline] }
[Pipeline] // container
[Pipeline] container
[Pipeline] {[Pipeline] sh
+ echo hello world
hello world
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS

声明式模板不从父模板继承。由于在阶段级声明的代理可以覆盖全局代理，隐式继承导致了混乱。
如果需要，您需要使用字段inheritFrom显式地声明继承。在下面的例子中，nested-pod将只包含maven容器。

pipeline {agent {kubernetes {yaml '''spec:containers:- name: golangimage: golang:1.16.5command:- sleepargs:- 99d'''}}stages {stage('Run maven') {agent {kubernetes {yaml '''spec:containers:- name: mavenimage: maven:3.8.1-jdk-8command:- sleepargs:- 99d'''}}steps {…}}}
}

7 Jenkins Pipeline 构建镜像实例

7.1 git 拉取仓库 & 构建镜像

创建docker 容器，挂载/var/run/docker.sock文件即可获得在容器内构建镜像的条件。在pipeline中我从github拉取一个demo尝试构建镜像。

podTemplate(namespace: "default",yaml: '''apiVersion: v1kind: Podspec:containers:- name: dockerimage: docker:19.03.1command:- sleepargs:- 99dvolumeMounts:- name: dockersockmountPath: /var/run/docker.sockvolumes:- name: dockersockhostPath:path: /var/run/docker.sock
''') {node(POD_LABEL) {stage('Build Docker image') {git 'https://github.com/Ghostwritten/kaniko-python-docker.git'container('docker') {sh 'docker build -t ghostwritten/kaniko-python-docker:v1.0.0 .'}}}
}

console output：

Started by user Jenkins Admin
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {[Pipeline] node
Created Pod: kubernetes default/docker-6-mjk3l-z7k7p-zkt08
Agent docker-6-mjk3l-z7k7p-zkt08 is provisioned from template docker_6-mjk3l-z7k7p
---
apiVersion: "v1"
kind: "Pod"
metadata:annotations:buildUrl: "http://jenkins.jenkins.svc.cluster.local:8080/job/docker/6/"runUrl: "job/docker/6/"labels:jenkins/jenkins-jenkins-agent: "true"jenkins/label-digest: "79a0f8e1675392aae2023abc0c58dbb40369e1eb"jenkins/label: "docker_6-mjk3l"name: "docker-6-mjk3l-z7k7p-zkt08"namespace: "default"
spec:containers:- args:- "99d"command:- "sleep"image: "docker:19.03.1"name: "docker"volumeMounts:- mountPath: "/var/run/docker.sock"name: "dockersock"- mountPath: "/home/jenkins/agent"name: "workspace-volume"readOnly: false- env:- name: "JENKINS_SECRET"value: "********"- name: "JENKINS_TUNNEL"value: "jenkins-agent.jenkins.svc.cluster.local:50000"- name: "JENKINS_AGENT_NAME"value: "docker-6-mjk3l-z7k7p-zkt08"- name: "JENKINS_NAME"value: "docker-6-mjk3l-z7k7p-zkt08"- name: "JENKINS_AGENT_WORKDIR"value: "/home/jenkins/agent"- name: "JENKINS_URL"value: "http://jenkins.jenkins.svc.cluster.local:8080/"image: "jenkins/inbound-agent:4.11-1-jdk11"name: "jnlp"resources:limits: {}requests:memory: "256Mi"cpu: "100m"volumeMounts:- mountPath: "/home/jenkins/agent"name: "workspace-volume"readOnly: falsenodeSelector:kubernetes.io/os: "linux"restartPolicy: "Never"volumes:- hostPath:path: "/var/run/docker.sock"name: "dockersock"- emptyDir:medium: ""name: "workspace-volume"Running on docker-6-mjk3l-z7k7p-zkt08 in /home/jenkins/agent/workspace/docker
[Pipeline] {[Pipeline] stage
[Pipeline] { (Build Docker image)
[Pipeline] git
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/Ghostwritten/kaniko-python-docker.git> git init /home/jenkins/agent/workspace/docker # timeout=10
Fetching upstream changes from https://github.com/Ghostwritten/kaniko-python-docker.git> git --version # timeout=10> git --version # 'git version 2.30.2'> git fetch --tags --force --progress -- https://github.com/Ghostwritten/kaniko-python-docker.git +refs/heads/*:refs/remotes/origin/* # timeout=10> git config remote.origin.url https://github.com/Ghostwritten/kaniko-python-docker.git # timeout=10> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision 925d4779eb7b4d41840d9daabb5ef5518d65ed1c (refs/remotes/origin/master)> git rev-parse refs/remotes/origin/master^{commit} # timeout=10> git config core.sparsecheckout # timeout=10> git checkout -f 925d4779eb7b4d41840d9daabb5ef5518d65ed1c # timeout=10> git branch -a -v --no-abbrev # timeout=10> git checkout -b master 925d4779eb7b4d41840d9daabb5ef5518d65ed1c # timeout=10
Commit message: "add kaniko python docker"> git rev-list --no-walk 925d4779eb7b4d41840d9daabb5ef5518d65ed1c # timeout=10
[Pipeline] container
[Pipeline] {[Pipeline] sh
+ docker build -t ghostwritten/kaniko-python-docker:v1.0.0 .
Sending build context to Docker daemon  82.43kBStep 1/6 : FROM python:3.8-slim-buster---> 5cc8cb0c433a
Step 2/6 : WORKDIR /app---> Using cache---> 58118ab4f40e
Step 3/6 : COPY requirements.txt requirements.txt---> Using cache---> 638813138f19
Step 4/6 : RUN pip3 install -r requirements.txt---> Using cache---> 9dd11465b0d8
Step 5/6 : COPY . .---> 4d92feff0580
Step 6/6 : CMD [ "python3", "app.py"]---> Running in fca5c52c6378
Removing intermediate container fca5c52c6378---> 048935596734
Successfully built 048935596734
Successfully tagged ghostwritten/kaniko-python-docker:v1.0.0
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS

构建成功。

7.2 编写 Dockerfile & 构建镜像

不过有时候我们直接在 Pipeline Script 中编写Dockerfile更加方便。

podTemplate(name: "testing", namespace: "default", yaml: '''apiVersion: v1kind: Podspec:volumes:- name: docker-sockhostPath: path: /var/run/docker.sockcontainers:- name: dockerimage: docker:19.03.1command:- sleepargs:- 99dvolumeMounts:- name: docker-sockmountPath: /var/run/docker.sock- name: docker-daemonimage: docker:19.03.1-dindsecurityContext:privileged: truevolumeMounts:- name: docker-sockmountPath: /var/run/docker.sock
''') {node(POD_LABEL) {writeFile file: 'Dockerfile', text: 'FROM scratch'container('docker') {sh 'docker version && DOCKER_BUILDKIT=1 docker build --progress plain -t testing .'}}
}

Pipeline Steps:

构建成功，太棒啦。