【云原生 | Kubernetes 系列】--持续交付和持续部署GITOPS(上)
2024-05-13 21:25:41
1. 持续交付和持续部署
Continuous Integration
Continuous Delivery
Continuous Deployment
Plan Code Build Test Release Deploy Operate
2. CI/CD Pipeline
为了交付新版本软件而必须执行的一系列步骤
一套专注于使用Devops或SRE方法来改进交付实践
加入了监控和自动化来改进应用开发过程,尤其是
3. Gitops工具集合
Git和Git server: Gitlab,Github
CI server: Jenkins,Tekton
Agent Operator: ArgoCD,Flux,Jenkins X,WKSctl和Gitkube等
Canary Deployer: Flux的Flagger等支持金丝雀发布,Istio,Linkerd,App mesh,NGIX,skipper,Contour,Gloo,Traefik等支持流量和路由和流量迁移,以及根据prometheus实现Canary分析
4. GitOps示例 Tekton and ArgoCD
Tekton实现CI Pipeline
代码克隆,单元测试,代码扫描,镜像构建,镜像推送
ArgoCD实现CD Pipeline
监视镜像变动,通过配置仓库拿到配置变更信息,同步到k8s环境上
5. Tekton 术语
1到多个step组成一个task
1到多个task组成一个pipeline
在同一个task中step依次运行
在同一个pipeline可以并行或串行运行
Step:
CI/CD工作流中的一个具体操作,每个step都会通过一个特定的Container运行
Task:
由一组step组成的序列,按照定义的顺序依次运行于同一个Pod内的不同容器中,可共享一组环境变量及存储卷
Pipeline:
由一组Task组成的集合,可按照定义比不同的方式运行:串行,并行,DAG
一个Task的输出可由其后Task引用
早起task和step之间共享是使用input和output resources进行共享
6. Parameters
Parameters是使得Task及Pipeline资源定义出的"模板"更加具有通用性的关键要素之一
将目标定义为参数(parameters),通过参数实例化传递给Task,而在step的代码中引用这些参数作为操作对象.TaskRun在针对Task进行实例化时,通过向引用Task中定义参数传值完成实例化.
7. Tekton系统组件
Tekton Pipelines
- tekton最核心的组件
- 必须运行在kubernetes集群上,作为k8s的扩展
Tekton Triggers
- 触发器,可触发Pipeline的实例化;可选组件
Tekton CLI
- 命令行客户端工具,用于与Tekton进行交互;可选组件
Tekton Dashboard
- Tekton Pipelines的基于web的图形界面;可选组件
Tekton Catalog
- 由社区贡献的Tekton构建块(building blocks),用户可直接使用
Tekton Hub
- 用于访问Catalog的图形界面,基于Web
8. 部署安装Tekton
Tekton
Tekton Pipeline
Task,TaskRun,Pipeline和PipelineRun等
Tekton Dashboard
Web UI
Tekton CLI
以命令行方式管理Pipelines的CRD所提供的API资源
8.1 获取镜像
通过阿里云的镜像服务获取到controller和webhook镜像,这里就忽略了.可以去之前的文档里找.如果懒得弄.直接用我的镜像也可以.harbor域名自己改一下就好了
root@k8s-master-01:~# docker pull registry.cn-shanghai.aliyuncs.com/qiuqin/tekton-controller:tekton-webhook
tekton-webhook: Pulling from qiuqin/tekton-controller
e0fe5e9ecdfc: Pull complete
befb56182419: Pull complete
e1ad716a7ed3: Pull complete
Digest: sha256:d2b731b6585763f2efeca08a185d4e0b24fbd3778580a9f7fdb425d7092c8aac
Status: Downloaded newer image for registry.cn-shanghai.aliyuncs.com/qiuqin/tekton-controller:tekton-webhook
registry.cn-shanghai.aliyuncs.com/qiuqin/tekton-controller:tekton-webhook
root@k8s-master-01:~# docker pull registry.cn-shanghai.aliyuncs.com/qiuqin/tekton-controller:tekton-controller
tekton-controller: Pulling from qiuqin/tekton-controller
e0fe5e9ecdfc: Already exists
befb56182419: Already exists
b85e63cd8929: Pull complete
Digest: sha256:f5f3c0012e70e6b673c0cb7b7b95c1811228a76ea345973b432789fa93d38f02
Status: Downloaded newer image for registry.cn-shanghai.aliyuncs.com/qiuqin/tekton-controller:tekton-controller
registry.cn-shanghai.aliyuncs.com/qiuqin/tekton-controller:tekton-controllerroot@k8s-master-01:~# docker tag registry.cn-shanghai.aliyuncs.com/qiuqin/tekton-controller:tekton-webhook gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/webhook:v0.40.2@sha256:6b8aadbdcede63969ecb719e910b55b7681d87110fc0bf92ca4ee943042f620b
refusing to create a tag with a digest reference
root@k8s-master-01:~# docker tag registry.cn-shanghai.aliyuncs.com/qiuqin/tekton-controller:tekton-webhook harbor.intra.com/tekton/webhook:v0.40.2
root@k8s-master-01:~# docker push harbor.intra.com/tekton/webhook:v0.40.2
The push refers to repository [harbor.intra.com/tekton/webhook]
bb9ab3658c2d: Pushed
f5a34658a837: Pushed
f860ed6c76e3: Pushed
v0.40.2: digest: sha256:08b4ef5f85a239e2a518ed5b4461006309c4b7ab45b1d81c33040064cea43721 size: 950
root@k8s-master-01:~# docker tag registry.cn-shanghai.aliyuncs.com/qiuqin/tekton-controller:tekton-controller harbor.intra.com/tekton/controller:v0.40.2
root@k8s-master-01:~# docker push harbor.intra.com/tekton/controller:v0.40.2
The push refers to repository [harbor.intra.com/tekton/controller]
a5a9958e61d5: Pushed
f5a34658a837: Mounted from tekton/webhook
f860ed6c76e3: Mounted from tekton/webhook
v0.40.2: digest: sha256:cb6907120bf98620b94b37d079ac92366c9b70fc17e864ec90ee91ee4fdd5e7c size: 9508.2 获取并修改yaml
下载release.yaml后修改2处image
root@k8s-master-01:~# wget https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
--2022-10-22 11:01:24-- https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
Resolving storage.googleapis.com (storage.googleapis.com)... 142.251.42.240, 172.217.160.112, 172.217.163.48
Connecting to storage.googleapis.com (storage.googleapis.com)|142.251.42.240|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 79895 (78K) [application/octet-stream]
Saving to: ‘release.yaml’release.yaml 100%[===================================================================>] 78.02K --.-KB/s in 0.1s 2022-10-22 11:01:25 (709 KB/s) - ‘release.yaml’ saved [79895/79895]
root@k8s-master-01:~# vi release.yaml
略...containers:- name: tekton-pipelines-controllerimage: harbor.intra.com/tekton/controller:v0.40.2#image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/controller:v0.40.2@sha256:dc7bc7d6607466b502d8dc22ba0598461d7477f608ab68aaff1ff4dedaa04f81
略...containers:- name: webhook# This is the Go import path for the binary that is containerized# and substituted here.image: harbor.intra.com/tekton/webhook:v0.40.2#image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/webhook:v0.40.2@sha256:6b8aadbdcede63969ecb719e910b55b7681d87110fc0bf92ca4ee943042f620broot@k8s-master-01:~# kubectl apply -f release.yaml
namespace/tekton-pipelines created
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/tekton-pipelines created
clusterrole.rbac.authorization.k8s.io/tekton-pipelines-controller-cluster-access created
clusterrole.rbac.authorization.k8s.io/tekton-pipelines-controller-tenant-access created
clusterrole.rbac.authorization.k8s.io/tekton-pipelines-webhook-cluster-access created
role.rbac.authorization.k8s.io/tekton-pipelines-controller created
role.rbac.authorization.k8s.io/tekton-pipelines-webhook created
role.rbac.authorization.k8s.io/tekton-pipelines-leader-election created
role.rbac.authorization.k8s.io/tekton-pipelines-info created
serviceaccount/tekton-pipelines-controller created
serviceaccount/tekton-pipelines-webhook created
clusterrolebinding.rbac.authorization.k8s.io/tekton-pipelines-controller-cluster-access created
clusterrolebinding.rbac.authorization.k8s.io/tekton-pipelines-controller-tenant-access created
clusterrolebinding.rbac.authorization.k8s.io/tekton-pipelines-webhook-cluster-access created
rolebinding.rbac.authorization.k8s.io/tekton-pipelines-controller created
rolebinding.rbac.authorization.k8s.io/tekton-pipelines-webhook created
rolebinding.rbac.authorization.k8s.io/tekton-pipelines-controller-leaderelection created
rolebinding.rbac.authorization.k8s.io/tekton-pipelines-webhook-leaderelection created
rolebinding.rbac.authorization.k8s.io/tekton-pipelines-info created
customresourcedefinition.apiextensions.k8s.io/clustertasks.tekton.dev created
customresourcedefinition.apiextensions.k8s.io/pipelines.tekton.dev created
customresourcedefinition.apiextensions.k8s.io/pipelineruns.tekton.dev created
customresourcedefinition.apiextensions.k8s.io/resolutionrequests.resolution.tekton.dev created
customresourcedefinition.apiextensions.k8s.io/pipelineresources.tekton.dev created
customresourcedefinition.apiextensions.k8s.io/runs.tekton.dev created
customresourcedefinition.apiextensions.k8s.io/tasks.tekton.dev created
customresourcedefinition.apiextensions.k8s.io/taskruns.tekton.dev created
secret/webhook-certs created
validatingwebhookconfiguration.admissionregistration.k8s.io/validation.webhook.pipeline.tekton.dev created
mutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.pipeline.tekton.dev created
validatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.pipeline.tekton.dev created
clusterrole.rbac.authorization.k8s.io/tekton-aggregate-edit created
clusterrole.rbac.authorization.k8s.io/tekton-aggregate-view created
configmap/config-artifact-bucket created
configmap/config-artifact-pvc created
configmap/config-defaults created
configmap/feature-flags created
configmap/pipelines-info created
configmap/config-leader-election created
configmap/config-logging created
configmap/config-observability created
configmap/config-registry-cert created
deployment.apps/tekton-pipelines-controller created
service/tekton-pipelines-controller created
Warning: autoscaling/v2beta1 HorizontalPodAutoscaler is deprecated in v1.22+, unavailable in v1.25+; use autoscaling/v2beta2 HorizontalPodAutoscaler
horizontalpodautoscaler.autoscaling/tekton-pipelines-webhook created
deployment.apps/tekton-pipelines-webhook created
service/tekton-pipelines-webhook created
执行后会创建一个tekton-pipelines命名空间,并在tekton-pipelines命名空间创建pods
root@k8s-master-01:~# kubectl get ns
NAME STATUS AGE
default Active 177d
ehelp Active 35d
hr Active 9d
istio-system Active 8d
kube-node-lease Active 177d
kube-public Active 177d
kube-system Active 177d
kubernetes-dashboard Active 83d
kuboard Active 82d
monitoring Active 52d
tekton-pipelines Active 25s
root@k8s-master-01:~# kubectl get pods -n tekton-pipelines
NAME READY STATUS RESTARTS AGE
tekton-pipelines-controller-54c78999-hp8fq 1/1 Running 0 33s
tekton-pipelines-webhook-7b8876dd96-ph264 1/1 Running 0 33s
root@k8s-master-01:~# kubectl api-resources --api-group=tekton.dev
NAME SHORTNAMES APIVERSION NAMESPACED KIND
clustertasks tekton.dev/v1beta1 false ClusterTask#集群级别的task
pipelineresources tekton.dev/v1alpha1 true PipelineResource#废弃
pipelineruns pr,prs tekton.dev/v1beta1 true PipelineRun
pipelines tekton.dev/v1beta1 true Pipeline
runs tekton.dev/v1alpha1 true Run # taskrun和pipeline的集合
taskruns tr,trs tekton.dev/v1beta1 true TaskRun
tasks tekton.dev/v1beta1 true Task
8.2 部署Tekton Dashboard
先下载yaml
root@k8s-master-01:~# wget https://storage.googleapis.com/tekton-releases/dashboard/latest/tekton-dashboard-release.yaml
--2022-10-22 20:55:37-- https://storage.googleapis.com/tekton-releases/dashboard/latest/tekton-dashboard-release.yaml
Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.160.112, 142.251.42.240, 142.251.43.16
Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.160.112|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8935 (8.7K) [application/octet-stream]
Saving to: ‘tekton-dashboard-release.yaml’tekton-dashboard-release.yaml 100%[===================================================================>] 8.73K --.-KB/s in 0.002s 2022-10-22 20:55:39 (3.81 MB/s) - ‘tekton-dashboard-release.yaml’ saved [8935/8935]
一样曲线下载到image并修改yaml
root@k8s-master-01:~# docker tag registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:dashboard harbor.intra.com/tekton/dashboaard:v0.29.2
root@k8s-master-01:~# docker push harbor.intra.com/tekton/dashboaard:v0.29.2
The push refers to repository [harbor.intra.com/tekton/dashboaard]
6da4284cc433: Pushed
02e4340fb2bf: Pushed
cacee284090d: Pushed
fa9b438e48f4: Pushed
v0.29.2: digest: sha256:413e678ce7d56f83d85af9ed873ad19f1b74008fd0474fb0d0d5b17acc1f3762 size: 1160
修改yaml中镜像
# vi tekton-dashboard-release.yaml
略..env:- name: INSTALLED_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespaceimage: harbor.intra.com/tekton/dashboaard:v0.29.2#image: gcr.io/tekton-releases/github.com/tektoncd/dashboard/cmd/dashboard:v0.29.2@sha256:67425f5fa32c591578ba6c8b6d4d6a135f8ca195745e023f3f311e3f1ee6ea37livenessProbe:部署dashboard
root@k8s-master-01:~# kubectl apply -f tekton-dashboard-release.yaml
customresourcedefinition.apiextensions.k8s.io/extensions.dashboard.tekton.dev created
serviceaccount/tekton-dashboard created
role.rbac.authorization.k8s.io/tekton-dashboard-info created
clusterrole.rbac.authorization.k8s.io/tekton-dashboard-backend created
clusterrole.rbac.authorization.k8s.io/tekton-dashboard-tenant created
rolebinding.rbac.authorization.k8s.io/tekton-dashboard-info created
clusterrolebinding.rbac.authorization.k8s.io/tekton-dashboard-backend created
configmap/dashboard-info created
service/tekton-dashboard created
deployment.apps/tekton-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/tekton-dashboard-tenant created
root@k8s-master-01:~# kubectl get pods -n tekton-pipelines
NAME READY STATUS RESTARTS AGE
tekton-dashboard-55654dbf76-gntt6 1/1 Running 0 64s
tekton-pipelines-controller-54c78999-hp8fq 1/1 Running 0 8h
tekton-pipelines-webhook-7b8876dd96-ph264 1/1 Running 0 8h
root@k8s-master-01:~# kubectl get svc -n tekton-pipelines
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
tekton-dashboard ClusterIP 10.200.158.50 <none> 9097/TCP 73s
tekton-pipelines-controller ClusterIP 10.200.56.104 <none> 9090/TCP,8008/TCP,8080/TCP 28h
tekton-pipelines-webhook ClusterIP 10.200.249.60 <none> 9090/TCP,8008/TCP,443/TCP,8080/TCP 28h8.3 部署istio
# istioctl apply --set profile=demo
This will install the Istio 1.13.3 demo profile with ["Istio core" "Istiod" "Ingress gateways" "Egress gateways"] components into the cluster. Proceed? (y/N) y
✔ Istio core installed
✔ Istiod installed
✔ Egress gateways installed
✔ Ingress gateways installed
✔ Installation complete Making this installation the default for injection and validation.Thank you for installing Istio 1.13. Please take a few minutes to tell us about your install/upgrade experience! https://forms.gle/pzWZpAvMVBecaQ9h9
# kubectl get svc -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana ClusterIP 10.200.58.210 <none> 3000/TCP 9d
istio-egressgateway ClusterIP 10.200.188.36 <none> 80/TCP,443/TCP 9d
istio-ingressgateway LoadBalancer 10.200.116.152 192.168.31.163 15021:36408/TCP,80:32291/TCP,443:46749/TCP,31400:60601/TCP,15443:57185/TCP,20001:52713/TCP 9d
istiod ClusterIP 10.200.241.72 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 9d
jaeger-collector ClusterIP 10.200.226.128 <none> 14268/TCP,14250/TCP,9411/TCP 9d
kiali ClusterIP 10.200.49.124 <none> 20001/TCP,9090/TCP 9d
prometheus ClusterIP 10.200.0.132 <none> 9090/TCP 9d
tracing ClusterIP 10.200.108.83 <none> 80/TCP,16685/TCP 9d
zipkin ClusterIP 10.200.136.249 <none> 9411/TCP 9d
8.4 将tekton用istio ingress对外提供服务
8.4.1 yaml配置
# cat 03-virtualservice-tekton-dashboard.yaml
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:name: tekton-dashboardnamespace: tekton-pipelines
spec:host: tekton-dashboardtrafficPolicy:tls:mode: DISABLE
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:name: tekton-dashboard-gatewaynamespace: istio-system
spec:selector:app: istio-ingressgatewayservers:- port:number: 80name: httpprotocol: HTTPhosts:- "tekton.intra.com"- "ci.intra.com"
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:name: tekton-dashboard-virtualservicenamespace: tekton-pipelines
spec:hosts:- "tekton.intra.com"- "ci.intra.com"gateways:- istio-system/tekton-dashboard-gatewayhttp:- match:- uri:prefix: /route:- destination:host: tekton-dashboardport:number: 9097
---
8.4.2 部署服务
# kubectl apply -f 03-virtualservice-tekton-dashboard.yaml
destinationrule.networking.istio.io/tekton-dashboard created
gateway.networking.istio.io/tekton-dashboard-gateway created
virtualservice.networking.istio.io/tekton-dashboard-virtualservice created
# kubectl get vs -n tekton-pipelines
NAME GATEWAYS HOSTS AGE
tekton-dashboard-virtualservice ["istio-system/tekton-dashboard-gateway"] ["tekton.intra.com","ci.intra.com"] 75s
测试访问
8.4 部署Tekton CLI
下载安装tekton CLI
root@k8s-master-01:~# wget https://github.com/tektoncd/cli/releases/download/v0.24.0/tektoncd-cli-0.24.0_Linux-64bit.deb
root@k8s-master-01:~# dpkg -i tektoncd-cli-0.24.0_Linux-64bit.deb
(Reading database ... 129211 files and directories currently installed.)
Preparing to unpack tektoncd-cli-0.24.0_Linux-64bit.deb ...
Unpacking cli (0.24.0) ...
Setting up cli (0.24.0) ...
root@k8s-master-01:~# tkn version
Client version: 0.24.0
Pipeline version: v0.40.2
Dashboard version: v0.29.2
9. 创建并运行一个task
9.1 task-hello配置文件
# cat 01-task-hello.yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:name: hello
spec:steps:- name: say-helloimage: alpine:3.15command: ['/bin/sh']args: ['-c', 'echo Hello World']
# kubectl apply -f 01-task-hello.yaml
task.tekton.dev/hello created
# tkn tasks list
NAME DESCRIPTION AGE
hello 15 minutes ago
9.2 启动并运行task
下载镜像并发布到其他node节点
# docker pull registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:entrypoint
# docker tag registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:entrypoint harbor.intra.com/tekton/entrypoint:v0.40.2
# docker push harbor.intra.com/tekton/entrypoint:v0.40.2
The push refers to repository [harbor.intra.com/tekton/entrypoint]
d7bb2e84b96d: Pushed
f5a34658a837: Mounted from tekton/controller
f860ed6c76e3: Mounted from tekton/controller
v0.40.2: digest: sha256:e9922fd8e8a9a71feb3541d4bac8ebdba289a48c0072e0109be2edfde1b2b7a9 size: 950
## 其他几个镜像
# docker pull registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:kubeconfigwriter
kubeconfigwriter: Pulling from qiuqin/tekton
e0fe5e9ecdfc: Already exists
befb56182419: Already exists
50347b3a0a2c: Pull complete
Digest: sha256:4c437956fcaf30b4801ead4087081301377104ebde37fd807f5a86d0e7d71487
Status: Downloaded newer image for registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:kubeconfigwriter
registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:kubeconfigwriter
# docker tag registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:kubeconfigwriter harbor.intra.com/tekton/kubeconfigwriter:v0.40.2
# docker push harbor.intra.com/tekton/kubeconfigwriter:v0.40.2
The push refers to repository [harbor.intra.com/tekton/kubeconfigwriter]
46fa8331cd80: Pushed
f5a34658a837: Mounted from tekton/entrypoint
f860ed6c76e3: Mounted from tekton/entrypoint
v0.40.2: digest: sha256:61eb966313c44ed42045684323f80dd8e18ad6e3f9677d47fcd5b66e575481a4 size: 950root@k8s-master-01:~# docker pull registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:nop
nop: Pulling from qiuqin/tekton
e0fe5e9ecdfc: Already exists
befb56182419: Already exists
a52cfffe6985: Pull complete
Digest: sha256:bc7ec42659f7e871b45d0942fde57dfe8f72e0736d2e7c1d3df1ee9dae0cd0ab
Status: Downloaded newer image for registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:nop
registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:nop
root@k8s-master-01:~# docker tag registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:nop harbor.intra.com/tekton/nop:v0.40.2
root@k8s-master-01:~# docker push harbor.intra.com/tekton/nop:v0.40.2
The push refers to repository [harbor.intra.com/tekton/nop]
af022ec0f9be: Pushed
f5a34658a837: Mounted from tekton/kubeconfigwriter
f860ed6c76e3: Mounted from tekton/kubeconfigwriter
v0.40.2: digest: sha256:87b1f985f5a09ec3a9361aa81312948750057ed4d313a97865a4f0b4738289e6 size: 949root@k8s-master-01:~# docker pull registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:git-init
git-init: Pulling from qiuqin/tekton
674ac02c6124: Pull complete
befb56182419: Pull complete
44b819ef9759: Pull complete
Digest: sha256:40f976c38ec0d0920c34828af690b8fd3fe6cf067fe3a1b7f6ce1f6a9771ee89
root@k8s-master-01:~# docker tag registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:git-init harbor.intra.com/tekton/git-init:v0.40.2
root@k8s-master-01:~# docker push harbor.intra.com/tekton/git-init:v0.40.2
The push refers to repository [harbor.intra.com/tekton/git-init]
3b0aed08b854: Pushed
f5a34658a837: Mounted from tekton/nop
8f45c3833da4: Pushed
v0.40.2: digest: sha256:f0d3b823640c7a8d75c0f89e327988e756e8ae292f855bc9b74643f6e1d732ec size: 952root@k8s-master-01:~# docker pull registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:imagedigestexporter
imagedigestexporter: Pulling from qiuqin/tekton
e0fe5e9ecdfc: Already exists
befb56182419: Already exists
ddb6a682c5b6: Pull complete
Digest: sha256:297a0e1c8b1b625b60812b269f5e1e91c13e39dd5073fb75522dc5605fa5554f
Status: Downloaded newer image for registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:imagedigestexporter
registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:imagedigestexporter
root@k8s-master-01:~# docker tag registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:imagedigestexporter harbor.intra.com/tekton/imagedigestexporter:v0.40.2
root@k8s-master-01:~# docker push harbor.intra.com/tekton/imagedigestexporter:v0.40.2
The push refers to repository [harbor.intra.com/tekton/imagedigestexporter]
203a83370407: Pushed
f5a34658a837: Mounted from tekton/git-init
f860ed6c76e3: Mounted from tekton/nop
v0.40.2: digest: sha256:90ca95be0b4907a6cf967bb932ab6494eec32d8d8321232633976da7cab832b6 size: 950root@k8s-master-01:~# docker pull registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:pullrequest-init
pullrequest-init: Pulling from qiuqin/tekton
e0fe5e9ecdfc: Already exists
befb56182419: Already exists
e50c9ddd4439: Pull complete
Digest: sha256:1c8f3d2845c01835da7ebba8e6a076b06d169aaa7b9c4760a56359107e1c3a37
Status: Downloaded newer image for registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:pullrequest-init
registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:pullrequest-init
root@k8s-master-01:~# docker tag registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:pullrequest-init harbor.intra.com/tekton/pullrequest-init:v0.40.2
root@k8s-master-01:~# docker push harbor.intra.com/tekton/pullrequest-init:v0.40.2
The push refers to repository [harbor.intra.com/tekton/pullrequest-init]
9f7b2b09045c: Pushed
f5a34658a837: Mounted from tekton/imagedigestexporter
f860ed6c76e3: Mounted from tekton/imagedigestexporter
v0.40.2: digest: sha256:5d8e2df4c108b9fe403bfaa966969623cc01184370d798d3f0c7b9310a206355 size: 949root@k8s-master-01:~# docker pull registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:workingdirinit
workingdirinit: Pulling from qiuqin/tekton
e0fe5e9ecdfc: Already exists
250c06f7c38e: Pull complete
319d199ee7b1: Pull complete
Digest: sha256:c7e327918fe23b3747409dba14a4a29c6165823273147f3558eb8eada0427435
Status: Downloaded newer image for registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:workingdirinit
registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:workingdirinit
root@k8s-master-01:~# docker tag registry.cn-shanghai.aliyuncs.com/qiuqin/tekton:workingdirinit harbor.intra.com/tekton/workingdirinit:v0.40.2
root@k8s-master-01:~# docker push harbor.intra.com/tekton/workingdirinit:v0.40.2
The push refers to repository [harbor.intra.com/tekton/workingdirinit]
7dca5b93db28: Pushed
ffe56a1c5f38: Pushed
f860ed6c76e3: Mounted from tekton/pullrequest-init
v0.40.2: digest: sha256:15e133137c6035a3db2527aa8c0d40f84b69d92d050200364de9bdabbdea1437 size: 945root@k8s-master-01:~# docker pull registry.cn-shanghai.aliyuncs.com/qiuqin/tekton2:busybox
busybox: Pulling from qiuqin/tekton2
365be80d8ba0: Pull complete
Digest: sha256:33743f62160962a713a4aadac8598a456e5718b8a9a6709e0f2d92742fea48a4
Status: Downloaded newer image for registry.cn-shanghai.aliyuncs.com/qiuqin/tekton2:busybox
registry.cn-shanghai.aliyuncs.com/qiuqin/tekton2:busybox
root@k8s-master-01:~# docker tag registry.cn-shanghai.aliyuncs.com/qiuqin/tekton2:busybox harbor.intra.com/tekton/busybox
root@k8s-master-01:~# docker push harbor.intra.com/tekton/busybox
Using default tag: latest
The push refers to repository [harbor.intra.com/tekton/busybox]
07a6174a568a: Pushed
latest: digest: sha256:158d1540de35f6bd66027950caf7f94c738bf0e92b32c3eca4484c71016806b5 size: 527修改release.yaml
把原来外网镜像都改为内网的,这样后续不会出现镜像无法拉取的情况.
containers:- name: tekton-pipelines-controllerimage: harbor.intra.com/tekton/controller:v0.40.2#image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/controller:v0.40.2@sha256:dc7bc7d6607466b502d8dc22ba0598461d7477f608ab68aaff1ff4dedaa04f81args: [# These images are built on-demand by `ko resolve` and are replaced# by image references by digest. "-kubeconfig-writer-image", "harbor.intra.com/tekton/kubeconfigwriter:v0.40.2", "-git-image", "harbor.intra.com/tekton/git-init:v0.40.2", "-entrypoint-image", "harbor.intra.com/tekton/entrypoint:v0.40.2", "-nop-image", "harbor.intra.com/tekton/nop:v0.40.2", "-imagedigest-exporter-image", "harbor.intra.com/tekton/imagedigestexporter:v0.40.2", "-pr-image", "harbor.intra.com/tekton/pullrequest-init:v0.40.2", "-workingdirinit-image", "harbor.intra.com/tekton/workingdirinit:v0.40.2",# This is gcr.io/google.com/cloudsdktool/cloud-sdk:302.0.0-slim"-gsutil-image", "harbor.intra.com/tekton/cloud-sdk",# The shell image must allow root in order to create directories and copy files to PVCs.# distroless.dev/busybox as of April 14 2022# image shall not contains tag, so it will be supported on a runtime like cri-o"-shell-image", "harbor.intra.com/tekton/busybox",# for script mode to work with windows we need a powershell image# pinning to nanoserver tag as of July 15 2021在重新部署下release.yaml
# kubectl apply -f release.yaml
执行taskrun
root@k8s-master-01:~# tkn task ls
NAME DESCRIPTION AGE
hello 12 hours ago
root@k8s-master-01:~# tkn task start hello
TaskRun started: hello-run-vdb9gIn order to track the TaskRun progress run:
tkn taskrun logs hello-run-vdb9g -f -n default
root@k8s-master-01:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-run-vdb9g-pod 0/1 Completed 0 12s
hello-run-wmzsr-r-hj6hk-pod 0/1 Completed 0 16m
hello-run-wmzsr-r-ljkcr-pod 0/1 Completed 0 4m27s
点开taskrun可以看到运行的结果
当再次运行taskrun时,会创建新的pod
# kubectl apply -f 01-taskrun-hello.yaml
taskrun.tekton.dev/hello-run-00001 created
root@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-run-00001-pod 0/1 Completed 0 8s
hello-run-vdb9g-pod 0/1 Completed 0 47h
hello-run-wmzsr-r-hj6hk-pod 0/1 Completed 0 47h
hello-run-wmzsr-r-ljkcr-pod 0/1 Completed 0 47h
9.3 定义params
9.3.1 创建Params-task
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:name: hello-params
spec:params: # 定义声明- name: target # 参数名称type: string # 参数的类型,只有2种,1.string(字符串) 2.array(数组)description: Name of somebody or something to greet # 参数简要描述default: ihaveapenihaveanapple # 参数的默认值steps: # 定义步骤- name: say-helloimage: alpine:3.15command:- /bin/shargs: ['-c', 'echo Hello $(params.target)'] # 引用参数值
创建TASK
# kubectl apply -f 02-task-with-params.yaml
task.tekton.dev/hello-params created
# tkn task ls
NAME DESCRIPTION AGE
hello 2 days ago
hello-params 2 minutes ago
9.3.2 实例化hello-params
如果不传入参数,默认就会按default值生效,将ihaveapenihaveanapple作为值传入
# tkn task start hello-params
? Value for param `target` of type `string`? (Default is `ihaveapenihaveanapple`) ihaveapenihaveanapple
TaskRun started: hello-params-run-nqdrgIn order to track the TaskRun progress run:
tkn taskrun logs hello-params-run-nqdrg -f -n default
当传入pineapple,默认值就不再生效
# tkn task start hello-params
? Value for param `target` of type `string`? (Default is `ihaveapenihaveanapple`) pineapple
TaskRun started: hello-params-run-t4dgcIn order to track the TaskRun progress run:
tkn taskrun logs hello-params-run-t4dgc -f -n default
也可以用-p直接进行参数传递
# tkn task start hello-params --showlog -p target=Tekton
TaskRun started: hello-params-run-nvfr5
Waiting for logs to be available...
[say-hello] Hello Tekton
9.4 在Step中运行脚本
script和command互斥
# cat 04-task-step-with-script.yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:name: script
spec:steps:- name: step-with-scriptimage: alpine:3.15script: |#!/bin/shecho "Step with Script..."echo "Installing necessary tooling"apk add curlcurl -s www.baidu.com && echo "Success" || echo "Fail" echo "All done!"
实例化task可以看到运行结果
# kubectl apply -f 04-task-step-with-script.yaml
task.tekton.dev/script created
root@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# tkn task ls
NAME DESCRIPTION AGE
hello 2 days ago
hello-params 51 minutes ago
script 6 seconds ago
root@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# tkn task start script
TaskRun started: script-run-48xldIn order to track the TaskRun progress run:
tkn taskrun logs script-run-48xld -f -n default
root@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# tkn task start script --showlog
TaskRun started: script-run-lvvv5
Waiting for logs to be available...
[step-with-script] Step with Script...
[step-with-script] Installing necessary tooling
[step-with-script] fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
[step-with-script] fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
[step-with-script] (1/5) Installing ca-certificates (20220614-r0)
[step-with-script] (2/5) Installing brotli-libs (1.0.9-r5)
[step-with-script] (3/5) Installing nghttp2-libs (1.46.0-r0)
[step-with-script] (4/5) Installing libcurl (7.80.0-r3)
[step-with-script] (5/5) Installing curl (7.80.0-r3)
[step-with-script] Executing busybox-1.34.1-r7.trigger
[step-with-script] Executing ca-certificates-20220614-r0.trigger
[step-with-script] OK: 8 MiB in 19 packages
[step-with-script] <!DOCTYPE html>
[step-with-script] <!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>使用百度前必读</a> <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a> 京ICP证030173号 <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>
[step-with-script] Success
[step-with-script] All done!
9.5 一个Task中多个step
当一个Task定义了多个step,一定是完成第一个后再完成第二个,第三个
当定义了多个step时,Task pod个数会是step的个数
# cat 03-task-multi-steps.yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:name: multiple
spec:steps:- name: firstimage: alpine:3.15command:- /bin/shargs: ['-c', 'echo First Step']- name: secondimage: alpine:3.15command:- /bin/shargs: ['-c', 'echo Second Step']- name: thirdimage: alpine:3.15command:- /bin/shargs: ['-c', 'echo Third Step']
实例化task
可以看到结果和我们预期是一样的.依次执行了3个step,pod的数量是3即使多次执行效果是一样的
# kubectl apply -f 03-task-multi-steps.yaml
task.tekton.dev/multiple created
root@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# tkn task ls
NAME DESCRIPTION AGE
hello 2 days ago
hello-params 1 hour ago
multiple 2 minutes ago
script 17 minutes ago
root@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# tkn task start multiple --showlog
TaskRun started: multiple-run-lxghd
Waiting for logs to be available...
[first] First Step[second] Second Step[third] Third Step
root@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# tkn task start multiple --showlog
TaskRun started: multiple-run-7lw52
Waiting for logs to be available...
[first] First Step[second] Second Step[third] Third Steproot@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# kubectl get pods|grep multiple
multiple-run-7lw52-pod 0/3 Completed 0 9s
multiple-run-lxghd-pod 0/3 Completed 0 5m25s
9.6 给脚本传递参数
# cat 05-task-script-and-parameters.yaml
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:name: logger
spec:params:- name: texttype: stringdescription: something to logdefault: "-"steps:- name: logimage: alpine:3.15script: |apk add -q tzdatacp /usr/share/zoneinfo/Asia/Shanghai /etc/localtimeDATETIME=$(date "+%F %T")echo [$DATETIME] - $(params.text)
实例化task
默认不传递参数时,使用default的值-,传递参数之后就将参数作为值在输出中显示
root@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# kubectl apply -f 05-task-script-and-parameters.yaml
task.tekton.dev/logger created
root@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# tkn task ls
NAME DESCRIPTION AGE
hello 2 days ago
hello-params 2 hours ago
logger 7 seconds ago
multiple 1 hour ago
script 1 hour ago
root@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# tkn task start logger --showlog
? Value for param `text` of type `string`? (Default is `-`) -
TaskRun started: logger-run-pjnj5
Waiting for logs to be available...
[log] [2022-10-25 12:39:48] - -root@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# tkn task start logger --showlog -p text="qiuqin is login"
TaskRun started: logger-run-t4296
Waiting for logs to be available...
[log] [2022-10-25 12:40:46] - qiuqin is login
9.7 定义pipeline和pipelinerun
- 定义task
- 定义pipeline引用task
- 执行pipeline运行为pipelinerun
PR负责实例化Pipeline,Pipeline负责实例化它引用的task为TaskRun
当没有定义Pipeline中Task顺序时,多个Task会并行执行.
# cat 06-pipeline-demo.yaml
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:name: pipeline-demo
spec:tasks:- name: first-tasktaskRef:name: hello- name: second-tasktaskRef:name: multiplerunAfter:- first-task
实例化pipeline
这里可以看到是按次序执行了2个task
root@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# kubectl apply -f 06-pipeline-demo.yaml
pipeline.tekton.dev/pipeline-demo created
root@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# tkn pipeline ls
NAME AGE LAST RUN STARTED DURATION STATUS
pipeline-demo 8 seconds ago --- --- --- ---
root@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# tkn pipeline start pipeline-demo --showlog
PipelineRun started: pipeline-demo-run-rlgzp
Waiting for logs to be available...
[first-task : say-hello] Hello World[second-task : first] First Step[second-task : second] Second Step[second-task : third] Third Step
9.7.1 DAG
当对pipeline修改后
# cat 06-pipeline-demo.yaml
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:name: pipeline-demo
spec:tasks:- name: first-tasktaskRef:name: hello- name: second-tasktaskRef:name: multiplerunAfter:- first-task- name: third-tasktaskRef:name: hello-paramsrunAfter:- first-task- name: fourth-tasktaskRef:name: loggerrunAfter:- second-task- third-task
root@k8s-master-01:/apps/tekton-and-argocd-in-practise/02-tekton-basics# tkn pipeline describe pipeline-demo
Name: pipeline-demo
Namespace: default
【云原生 | Kubernetes 系列】--持续交付和持续部署GITOPS(上)相关推荐
- 【云原生 | Kubernetes 系列】---Skywalking部署和监控
[云原生 | Kubernetes 系列]-Skywalking部署和监控 1. 分布式链路追踪概念 在较大的web集群和微服务环境中,客户端的一次请求可能需要经过多个不同的模块,多个不同中间件,多个 ...
- 【云原生Kubernetes系列第五篇】kubeadm v1.20 部署K8S 集群架构（人生这道选择题，总会有遗憾）
系列文章目录 ??即日起,更新云原生相关知识,喜欢的小伙伴可以给我点个三连喔 ??收录至专栏 云原生 ??[云原生Docker系列第一篇]Docker镜像管理 ??[云原生Docker系列第二篇]Do ...
- 【云原生 | Kubernetes 系列】--Gitops持续交付 Tekton Pipeline使用进阶(pvc和Results)
Tekton Pipeline使用进阶 1. 基于Maven项目构建 主要实现source to package的过程 包含2个Task: fetch-from-source和build 使用pvc实 ...
- 【云原生 | Kubernetes 系列】--Gitops持续交付和持续Tekton Triggers
Tekton Triggers Tekton 几个项目 Tekton Pipelines Task Steps:执行具体操作,在容器中执行的命令或脚本 command script TaskRun 运 ...
- 【云原生 | Kubernetes 系列】K8s 实战 一文学会如何从 PodSecurityPolicy 迁移到内置的 PodSecurity 准入控制器
PodSecurityPolicy 迁移到内置的 PodSecurity 准入控制器 前言 一.Pod 安全性 二.访问权限 三.标准化 PodSecurityPolicy 3.1.去掉纯粹变更性质的 ...
- 【云原生 | Kubernetes 系列】---AlertManager安装及告警发送
AlertManager安装及告警发送 Prometheus触发一条告警的过程 prometheus-> 触发阈值->超出持续时间->AlertManager->分组|抑制|静 ...
- 【云原生 | Kubernetes 系列】1个POD2个container实现Wordpress K8s部署
1. Wordpress架构 2. 构建wordpress镜像 2.1 构建nginx-base-wordpress镜像 Dockerfile FROM harbor.intra.com/baseim ...
- 【云原生 | Kubernetes 系列】----污点与容忍
污点与容忍 污点(taints),用于node节点排斥Pod调度.与亲和效果相反.即taint的node排斥Pod的创建. 容忍(toleration),用于Pod容忍Node节点的污点信息.即nod ...
- 【云原生 | Kubernetes 系列】K8s 实战 管理 Secret 详解
kubectl 管理 Secret 前言 一.使用 kubectl 管理 Secret 1.1.创建 Secret 1.2.验证 Secret 1.3.解码 Secret 1.4.清理 二.使用配置文 ...
最新文章
- 二十五、求单点的最短路径
- 简陋,山寨,Everything,桌面搜索,原理，源码
- 中职生计算机专业600分,来了！超全盘点高职分类中500-600分及以上的高中生能报的专业和院校名单！...
- MyEclipse/Eclipse 中使用javap
- python分支结构说课_Python_3.8平台上的分支结构（模块.类.函数）_11
- 【转】CSS3 圆角 阴影 渐变 透明 旋转等功能详述
- 深度理解__proto__ 和 prototype
- 缓冲区是人为设定的吗_人为的，但这真的是情报吗？
- QCA9531修改寄存器值控制GPIO
- 【编译原理】理解BNF
- 同比、环比的区别及计算公式
- cfree mysql_如何配置CFree才能开发MySql数据库应用
- ERNIE-Enhanced Language Representation with Informative Entities 阅读笔记
- 中国石油大学远程教育《大学英语（四）》第二阶段在线作业
- Error:Execution failed for task ':recordlib:lint'. Lint found errors in the project; aborting buil
- Docker部署Jenkins服务
- oracle rac 心跳参数 misscount disktimeout
- jsp实现数据提交以及jsp数据保存到本地
- Android scrollTo() scrollBy() Scroller讲解及应用
- 6-2对象作为数据成员
热门文章
- 智能养殖监控系统实现蛋鸡养殖规模化
- linux 终端 screem,Linux screen实操指南
- CSS学习笔记之浮动布局
- 用户界面交互设计的八项黄金法则
- A Survey on Neural Network Interpretability (神经网络的可解释性研究综述)
- 将npz文件转化为txt文件中出现错误（ValueError: Expected 1D or 2D array, got 3D array instead）的解决办法
- CAD安装完桌面没图标
- code block怎样导入整个文件夹_利用ArcGis导入数据之四（dxf——Geomap、CAD、CorelDraw）...
- CCF考试——201403-1相反数
- 史玉柱背后的女人——程晨