Kubernetes CKA认证运维工程师笔记-Kubernetes应用程序生命周期管理
2024-06-12 13:33:11
Kubernetes CKA认证运维工程师笔记-Kubernetes应用程序生命周期管理
- 1. 在Kubernetes中部署应用流程
- 2. 使用Deployment部署Java应用
- 2.1 Pod与Deployment的关系
- 2.2 Deployment控制器介绍
- 3. 服务编排（YAML）
- 3.1 服务编排：YAML文件创建资源对象
- 3.2 服务编排：YAML文件格式说明
- 3.3 服务编排：资源字段太多，记不住怎么办
- 4. 应用升级、弹性收缩、回滚、删除
- 5. 滚动升级与回滚实现机制
- 6. Pod对象：Pod设计思想、应用自修复、Init container、静态Pod
- 6.1 Pod对象：基本概念
- 6.2 Pod对象：存在的意义
- 6.3 Pod对象：容器分类
- 6.4 Pod对象：应用自恢复（重启策略+健康检查）
- 6.5 Pod对象：Init container
- 6.6 Pod对象：静态Pod
1. 在Kubernetes中部署应用流程
镜像基本分为三类：
- 基础镜像
- 运行环境镜像
- 项目镜像
2. 使用Deployment部署Java应用
0、制作镜像
1、使用Deployment控制器部署镜像
kubectl create deployment web --image=lizhenliang/java-demo
kubectl get deploy,pods
2、使用Service发布Pod
kubectl expose deployment web --port=80 --type=NodePort --target-port=8080 --name=web
kubectl get service
2.1 Pod与Deployment的关系
Deployment是最为常用的controllers（也称为workload），其他控制器还有DaemonSet、StatefulSet等。
controllers作用：
- 管理Pod对象
- 使用标签与Pod关联
- 控制器了Pod的运维，例如滚动更新、伸缩、副本管理、维护Pod状态等
2.2 Deployment控制器介绍
Deployment的功能：
- 管理Pod和ReplicaSet
- 具有上线部署、副本设定、滚动升级、回滚等功能
- 提供声明式更新，例如只更新一个新的Image
应用场景：网站、API、微服务
3. 服务编排（YAML）
3.1 服务编排：YAML文件创建资源对象
定义Deployment：
# 控制器定义
apiVersion: apps/v1
kind: Deployment
metadata:name: nginx-deploymentnamespace: default
spec:replicas: 3selector:matchLabels:app: nginx
# 被控制对象
template:metadata:labels:app: nginxspec:containers:- name: nginximage: nginx:1.19ports:- containerPort: 80
[root@k8s-master ~]# vim deployment.yaml
[root@k8s-master ~]# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: nginx-deploymentlabels:app: nginx
spec:replicas: 3selector:matchLabels:app: nginxtemplate:metadata:labels:app: nginxspec:containers:- name: nginximage: nginx:1.19ports:- containerPort: 80
[root@k8s-master ~]# kubectl apply -f deployment.yaml
deployment.apps/nginx-deployment created
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-dep-5f8dfc8c78-dvxp8 1/1 Running 2 6d8h
my-dep-5f8dfc8c78-f4ln4 1/1 Running 2 6d8h
my-dep-5f8dfc8c78-j9fqp 1/1 Running 2 6d8h
nginx-deployment-7cf55fb7bb-d2t9n 0/1 ContainerCreating 0 14s
nginx-deployment-7cf55fb7bb-gbtfr 0/1 ContainerCreating 0 14s
nginx-deployment-7cf55fb7bb-npfqz 0/1 ContainerCreating 0 14s
nginx-php 1/1 Running 1 47h
web-96d5df5c8-ghb6g 1/1 Running 2 9d
web2 1/1 Running 1 46h
[root@k8s-master ~]# kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
my-dep 3/3 3 3 6d8h
nginx-deployment 2/3 3 2 37s
web 1/1 1 1 9d
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-dep-5f8dfc8c78-dvxp8 1/1 Running 2 6d8h
my-dep-5f8dfc8c78-f4ln4 1/1 Running 2 6d8h
my-dep-5f8dfc8c78-j9fqp 1/1 Running 2 6d8h
nginx-deployment-7cf55fb7bb-d2t9n 1/1 Running 0 54s
nginx-deployment-7cf55fb7bb-gbtfr 1/1 Running 0 54s
nginx-deployment-7cf55fb7bb-npfqz 1/1 Running 0 54s
nginx-php 1/1 Running 1 47h
web-96d5df5c8-ghb6g 1/1 Running 2 9d
web2 1/1 Running 1 46h
[root@k8s-master ~]# kubectl get pods --show-labels
NAME READY STATUS RESTARTS AGE LABELS
my-dep-5f8dfc8c78-dvxp8 1/1 Running 2 6d8h app=my-dep,pod-template-hash=5f8dfc8c78
my-dep-5f8dfc8c78-f4ln4 1/1 Running 2 6d8h app=my-dep,pod-template-hash=5f8dfc8c78
my-dep-5f8dfc8c78-j9fqp 1/1 Running 2 6d8h app=my-dep,pod-template-hash=5f8dfc8c78
nginx-deployment-7cf55fb7bb-d2t9n 1/1 Running 0 2m57s app=nginx,pod-template-hash=7cf55fb7bb
nginx-deployment-7cf55fb7bb-gbtfr 1/1 Running 0 2m57s app=nginx,pod-template-hash=7cf55fb7bb
nginx-deployment-7cf55fb7bb-npfqz 1/1 Running 0 2m57s app=nginx,pod-template-hash=7cf55fb7bb
nginx-php 1/1 Running 1 47h run=nginx-php
web-96d5df5c8-ghb6g 1/1 Running 2 9d app=web,pod-template-hash=96d5df5c8
web2 1/1 Running 1 46h <none>[root@k8s-master ~]# vim service.yaml
[root@k8s-master ~]# cat service.yaml
apiVersion: v1
kind: Service
metadata:name: my-service
spec:selector:app: nginxports:- protocol: TCPport: 80targetPort: 80type: NodePort
[root@k8s-master ~]# kubectl apply -f service.yaml
service/my-service created
[root@k8s-master ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9d
my-dep NodePort 10.111.199.51 <none> 80:31734/TCP 6d8h
my-service NodePort 10.100.228.0 <none> 80:32433/TCP 12s
web NodePort 10.96.132.243 <none> 80:31340/TCP 9d
[root@k8s-master ~]# kubectl get ep
NAME ENDPOINTS AGE
kubernetes 10.0.0.61:6443 9d
my-dep 10.244.169.150:8080,10.244.36.84:8080,10.244.36.85:8080 6d8h
my-service 10.244.169.151:80,10.244.169.152:80,10.244.36.88:80 26s
web 10.244.36.86:80 9d
[root@k8s-master ~]# curl 10.100.228.0
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>body {width: 35em;margin: 0 auto;font-family: Tahoma, Verdana, Arial, sans-serif;}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-dep-5f8dfc8c78-dvxp8 1/1 Running 2 6d8h
my-dep-5f8dfc8c78-f4ln4 1/1 Running 2 6d8h
my-dep-5f8dfc8c78-j9fqp 1/1 Running 2 6d8h
nginx-deployment-7cf55fb7bb-d2t9n 1/1 Running 0 9m34s
nginx-deployment-7cf55fb7bb-gbtfr 1/1 Running 0 9m34s
nginx-deployment-7cf55fb7bb-npfqz 1/1 Running 0 9m34s
nginx-php 1/1 Running 1 47h
web-96d5df5c8-ghb6g 1/1 Running 2 9d
web2 1/1 Running 1 46h
[root@k8s-master ~]# kubectl logs nginx-deployment-7cf55fb7bb-d2t9n
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
[root@k8s-master ~]# kubectl logs nginx-deployment-7cf55fb7bb-gbtfr
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
[root@k8s-master ~]# kubectl logs nginx-deployment-7cf55fb7bb-npfqz
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
10.244.235.192 - - [01/Dec/2021:06:08:41 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
# 在浏览器里面输入IP地址加端口号
[root@k8s-master ~]# kubectl logs nginx-deployment-7cf55fb7bb-d2t9n
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
10.244.235.192 - - [01/Dec/2021:06:12:32 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:94.0) Gecko/20100101 Firefox/94.0" "-"
2021/12/01 06:12:32 [error] 32#32: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 10.244.235.192, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "10.0.0.61:32433", referrer: "http://10.0.0.61:32433/"
10.244.235.192 - - [01/Dec/2021:06:12:32 +0000] "GET /favicon.ico HTTP/1.1" 404 154 "http://10.0.0.61:32433/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:94.0) Gecko/20100101 Firefox/94.0" "-"
[root@k8s-master ~]# kubectl logs nginx-deployment-7cf55fb7bb-gbtfr
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
[root@k8s-master ~]# kubectl logs nginx-deployment-7cf55fb7bb-npfqz
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
10.244.235.192 - - [01/Dec/2021:06:08:41 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"[root@k8s-master ~]# kubectl api-versions
admissionregistration.k8s.io/v1
admissionregistration.k8s.io/v1beta1
apiextensions.k8s.io/v1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1
apiregistration.k8s.io/v1beta1
apps/v1
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2beta1
autoscaling/v2beta2
batch/v1
batch/v1beta1
certificates.k8s.io/v1
certificates.k8s.io/v1beta1
coordination.k8s.io/v1
coordination.k8s.io/v1beta1
crd.projectcalico.org/v1
discovery.k8s.io/v1beta1
events.k8s.io/v1
events.k8s.io/v1beta1
extensions/v1beta1
metrics.k8s.io/v1beta1
networking.k8s.io/v1
networking.k8s.io/v1beta1
node.k8s.io/v1beta1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
scheduling.k8s.io/v1
scheduling.k8s.io/v1beta1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1
标签在资源关联的用法：
- 标签选择器 selector
- 标签
| 项目 | 解释 |
|---|---|
| apiVersion | API版本 |
| kind | 资源类型 |
| metadata | 资源元数据 |
| spec | 资源规格 |
| replicas | 副本数量 |
| selector | 标签选择器 |
| template | Pod模板 |
| metadata | Pod元数据 |
| spec | Pod规格 |
| containers | 容器配置 |
3.2 服务编排：YAML文件格式说明
YAML 是一种简洁的非标记语言。
语法格式：
- 缩进表示层级关系
- 不支持制表符 “tab” 缩进，使用空格缩进
- 通常开头缩进 2 个空格
- 字符后缩进 1 个空格，如冒号、逗号等
- “- - -” 表示YAML格式，一个文件的开始
- “#” 注释
3.3 服务编排：资源字段太多，记不住怎么办
- 用create命令生成
kubectl create deployment nginx --image=nginx:1.16 -o yaml --dry-run=client > my-deploy.yaml
- 用get命令导出
kubectl get deployment nginx -o yaml > my-deploy.yaml
- Pod容器的字段拼写忘记了
kubectl explain pods.spec.containers
kubectl explain deployment
-o, --output=’’: 输出指定的格式，常用的有json/yaml/name
[root@k8s-master ~]# kubectl create deployment nginx --image=nginx:1.16 -o yaml --dry-run=client
apiVersion: apps/v1
kind: Deployment
metadata:creationTimestamp: nulllabels:app: nginxname: nginx
spec:replicas: 1selector:matchLabels:app: nginxstrategy: {}template:metadata:creationTimestamp: nulllabels:app: nginxspec:containers:- image: nginx:1.16name: nginxresources: {}
status: {}[root@k8s-master ~]# kubectl get deploy web -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:annotations:deployment.kubernetes.io/revision: "1"creationTimestamp: "2021-11-22T03:30:03Z"generation: 1labels:app: webmanagedFields:- apiVersion: apps/v1fieldsType: FieldsV1fieldsV1:f:metadata:f:labels:.: {}f:app: {}f:spec:f:progressDeadlineSeconds: {}f:replicas: {}f:revisionHistoryLimit: {}f:selector:f:matchLabels:.: {}f:app: {}f:strategy:f:rollingUpdate:.: {}f:maxSurge: {}f:maxUnavailable: {}f:type: {}f:template:f:metadata:f:labels:.: {}f:app: {}f:spec:f:containers:k:{"name":"nginx"}:.: {}f:image: {}f:imagePullPolicy: {}f:name: {}f:resources: {}f:terminationMessagePath: {}f:terminationMessagePolicy: {}f:dnsPolicy: {}f:restartPolicy: {}f:schedulerName: {}f:securityContext: {}f:terminationGracePeriodSeconds: {}manager: kubectl-createoperation: Updatetime: "2021-11-22T03:30:03Z"- apiVersion: apps/v1fieldsType: FieldsV1fieldsV1:f:metadata:f:annotations:.: {}f:deployment.kubernetes.io/revision: {}f:status:f:conditions:.: {}k:{"type":"Available"}:.: {}f:lastTransitionTime: {}f:lastUpdateTime: {}f:message: {}f:reason: {}f:status: {}f:type: {}k:{"type":"Progressing"}:.: {}f:lastTransitionTime: {}f:lastUpdateTime: {}f:message: {}f:reason: {}f:status: {}f:type: {}f:observedGeneration: {}f:replicas: {}f:unavailableReplicas: {}f:updatedReplicas: {}manager: kube-controller-manageroperation: Updatetime: "2021-12-09T21:15:24Z"name: webnamespace: defaultresourceVersion: "1568972"selfLink: /apis/apps/v1/namespaces/default/deployments/webuid: ba0bbdcc-5760-4dce-86af-7eb647156b68
spec:progressDeadlineSeconds: 600replicas: 1revisionHistoryLimit: 10selector:matchLabels:app: webstrategy:rollingUpdate:maxSurge: 25%maxUnavailable: 25%type: RollingUpdatetemplate:metadata:creationTimestamp: nulllabels:app: webspec:containers:- image: nginximagePullPolicy: Alwaysname: nginxresources: {}terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilednsPolicy: ClusterFirstrestartPolicy: AlwaysschedulerName: default-schedulersecurityContext: {}terminationGracePeriodSeconds: 30
status:conditions:- lastTransitionTime: "2021-11-22T03:30:03Z"lastUpdateTime: "2021-11-22T03:30:28Z"message: ReplicaSet "web-96d5df5c8" has successfully progressed.reason: NewReplicaSetAvailablestatus: "True"type: Progressing- lastTransitionTime: "2021-12-09T21:15:24Z"lastUpdateTime: "2021-12-09T21:15:24Z"message: Deployment does not have minimum availability.reason: MinimumReplicasUnavailablestatus: "False"type: AvailableobservedGeneration: 1replicas: 1unavailableReplicas: 1updatedReplicas: 1
[root@k8s-master ~]# kubectl get deploy web -o yaml > deploy2.yaml
[root@k8s-master ~]# vi deploy2.yaml
[root@k8s-master ~]# cat deploy2.yaml
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: webname: webnamespace: default
spec:replicas: 1selector:matchLabels:app: webstrategy:rollingUpdate:maxSurge: 25%maxUnavailable: 25%type: RollingUpdatetemplate:metadata:labels:app: webspec:containers:- image: nginximagePullPolicy: Alwaysname: nginxresources: {}dnsPolicy: ClusterFirstrestartPolicy: Always
4. 应用升级、弹性收缩、回滚、删除
应用升级
kubectl set image deployment/web nginx=nginx:1.16
# 查看升级状态
kubectl rollout status deployment/web
[root@k8s-master ~]# kubectl set image deployment nginx-deployment nginx=nginx:1.16
deployment.apps/nginx-deployment image updated
[root@k8s-master ~]# kubectl get pods -w
NAME READY STATUS RESTARTS AGE
my-dep-5f8dfc8c78-dvxp8 1/1 Running 4 14d
my-dep-5f8dfc8c78-f4ln4 1/1 Running 4 14d
my-dep-5f8dfc8c78-j9fqp 1/1 Running 4 14d
nginx-deployment-644599b9c9-96ng4 0/1 ContainerCreating 0 4s
nginx-deployment-644599b9c9-c47jg 1/1 Running 0 31s
nginx-deployment-7cf55fb7bb-d2t9n 1/1 Running 2 8d
nginx-deployment-7cf55fb7bb-gbtfr 1/1 Running 2 8d
nginx-deployment-7cf55fb7bb-npfqz 0/1 Terminating 2 8d
nginx-php 1/1 Running 3 10d
web-96d5df5c8-ghb6g 1/1 Running 4 17d
web2 1/1 Running 3 10d
nginx-deployment-7cf55fb7bb-npfqz 0/1 Terminating 2 8d
nginx-deployment-7cf55fb7bb-npfqz 0/1 Terminating 2 8d
···
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-dep-5f8dfc8c78-dvxp8 1/1 Running 4 14d
my-dep-5f8dfc8c78-f4ln4 1/1 Running 4 14d
my-dep-5f8dfc8c78-j9fqp 1/1 Running 4 14d
nginx-deployment-644599b9c9-96ng4 1/1 Running 0 77s
nginx-deployment-644599b9c9-c47jg 1/1 Running 0 104s
nginx-deployment-644599b9c9-d45qx 1/1 Running 0 46s
nginx-php 1/1 Running 3 10d
web-96d5df5c8-ghb6g 1/1 Running 4 17d
web2 1/1 Running 3 10d
弹性伸缩
# 手动扩容
kubectl scale deployment web --replicas=10
# 自动水平扩容（注意Pod必须配置resource.request）
kubectl autoscale deployment web --min=3 --max=10 --cpu-percent=80
kubectl get hpa
[root@k8s-master ~]# kubectl scale deployment nginx-deployment --replicas=10
deployment.apps/nginx-deployment scaled
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-dep-5f8dfc8c78-dvxp8 1/1 Running 4 14d
my-dep-5f8dfc8c78-f4ln4 1/1 Running 4 14d
my-dep-5f8dfc8c78-j9fqp 1/1 Running 4 14d
nginx-deployment-644599b9c9-2ffhg 0/1 ContainerCreating 0 2s
nginx-deployment-644599b9c9-2t5vn 0/1 ContainerCreating 0 2s
nginx-deployment-644599b9c9-8tbgl 0/1 ContainerCreating 0 2s
nginx-deployment-644599b9c9-96ng4 1/1 Running 0 2m45s
nginx-deployment-644599b9c9-c47jg 1/1 Running 0 3m12s
nginx-deployment-644599b9c9-d45qx 1/1 Running 0 2m14s
nginx-deployment-644599b9c9-dhjzm 0/1 ContainerCreating 0 2s
nginx-deployment-644599b9c9-pvpwm 0/1 ContainerCreating 0 2s
nginx-deployment-644599b9c9-xsr8k 0/1 ContainerCreating 0 2s
nginx-deployment-644599b9c9-zqskd 0/1 ContainerCreating 0 2s
nginx-php 1/1 Running 3 10d
web-96d5df5c8-ghb6g 1/1 Running 4 17d
web2 1/1 Running 3 10d
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-dep-5f8dfc8c78-dvxp8 1/1 Running 4 14d
my-dep-5f8dfc8c78-f4ln4 1/1 Running 4 14d
my-dep-5f8dfc8c78-j9fqp 1/1 Running 4 14d
nginx-deployment-644599b9c9-2ffhg 1/1 Running 0 82s
nginx-deployment-644599b9c9-2t5vn 1/1 Running 0 82s
nginx-deployment-644599b9c9-8tbgl 1/1 Running 0 82s
nginx-deployment-644599b9c9-96ng4 1/1 Running 0 4m5s
nginx-deployment-644599b9c9-c47jg 1/1 Running 0 4m32s
nginx-deployment-644599b9c9-d45qx 1/1 Running 0 3m34s
nginx-deployment-644599b9c9-dhjzm 1/1 Running 0 82s
nginx-deployment-644599b9c9-pvpwm 1/1 Running 0 82s
nginx-deployment-644599b9c9-xsr8k 1/1 Running 0 82s
nginx-deployment-644599b9c9-zqskd 1/1 Running 0 82s
nginx-php 1/1 Running 3 10d
web-96d5df5c8-ghb6g 1/1 Running 4 17d
web2 1/1 Running 3 10dHPA：Pod水平扩容
kubectl top/hpa -> apiserver -> metrics-server -> kubelet -> pod
如果hpa出现unkunow：
- kubectl top 正常否
- deployment 是否配置 request
containers:- name: nginx
image: nginx:1.14.2
ports:- containerPort: 80
resources:
requests:
cpu: 0.5
- containerPort: 80
- name: nginx
[root@k8s-master ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-97769f7c7-z6npb 1/1 Running 4 18d
calico-node-4pwdc 1/1 Running 4 18d
calico-node-9r6zd 1/1 Running 4 18d
calico-node-vqzdj 1/1 Running 4 18d
coredns-6d56c8448f-gcgrh 1/1 Running 4 18d
coredns-6d56c8448f-tbsmv 1/1 Running 4 18d
etcd-k8s-master 1/1 Running 5 18d
kube-apiserver-k8s-master 1/1 Running 11 18d
kube-controller-manager-k8s-master 1/1 Running 13 17d
kube-proxy-5qpgc 1/1 Running 4 18d
kube-proxy-q2xfq 1/1 Running 4 18d
kube-proxy-tvzpd 1/1 Running 4 18d
kube-scheduler-k8s-master 1/1 Running 15 17d
metrics-server-84f9866fdf-kt2nb 1/1 Running 4 10d
[root@k8s-master ~]# kubectl api-resources|grep hpa
horizontalpodautoscalers hpa autoscaling true HorizontalPodAutoscaler[root@k8s-master ~]# kubectl autoscale deployment nginx-deployment --min=3 --max=10 --cpu-percent=10
horizontalpodautoscaler.autoscaling/nginx-deployment autoscaled
[root@k8s-master ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-deployment Deployment/nginx-deployment <unknown>/10% 3 10 10 40s
[root@k8s-master ~]# vi deployment.yaml
[root@k8s-master ~]# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: nginx-deploymentlabels:app: nginx
spec:replicas: 3selector:matchLabels:app: nginxtemplate:metadata:labels:app: nginxspec:containers:- name: nginximage: nginx:1.19ports:- containerPort: 80resources:requests:cpu: 0.5
[root@k8s-master ~]# kubectl apply -f deployment.yaml
deployment.apps/nginx-deployment configured
[root@k8s-master ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-deployment Deployment/nginx-deployment 0%/10% 3 10 3 8m28s# 对项目压测
[root@k8s-node1 ~]# yum install httpd-tools
[root@k8s-node1 ~]# ab -n 100000 -c 1000 http://10.100.228.0/index.html
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/Benchmarking 10.100.228.0 (be patient)
Completed 10000 requests
Completed 20000 requests[root@k8s-master ~]# kubectl top pod
NAME CPU(cores) MEMORY(bytes)
my-dep-5f8dfc8c78-dvxp8 3m 178Mi
my-dep-5f8dfc8c78-f4ln4 2m 189Mi
my-dep-5f8dfc8c78-j9fqp 2m 172Mi
nginx-deployment-7f78c49b8f-6fnq2 246m 4Mi
nginx-deployment-7f78c49b8f-ftzb7 395m 5Mi
nginx-deployment-7f78c49b8f-tjldz 267m 2Mi
nginx-php 1m 14Mi
web-96d5df5c8-ghb6g 0m 3Mi
web2 1m 16Mi
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-dep-5f8dfc8c78-dvxp8 1/1 Running 4 15d
my-dep-5f8dfc8c78-f4ln4 1/1 Running 4 15d
my-dep-5f8dfc8c78-j9fqp 1/1 Running 4 15d
nginx-deployment-7f78c49b8f-6fnq2 1/1 Running 0 6m40s
nginx-deployment-7f78c49b8f-ftzb7 1/1 Running 0 6m37s
nginx-deployment-7f78c49b8f-mjjkt 1/1 Running 0 10s
nginx-deployment-7f78c49b8f-mw5vb 1/1 Running 0 10s
nginx-deployment-7f78c49b8f-tjldz 1/1 Running 0 6m35s
nginx-deployment-7f78c49b8f-tr7v6 1/1 Running 0 10s
nginx-php 1/1 Running 3 10d
web-96d5df5c8-ghb6g 1/1 Running 4 17d
web2 1/1 Running 3 10d
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-dep-5f8dfc8c78-dvxp8 1/1 Running 4 15d
my-dep-5f8dfc8c78-f4ln4 1/1 Running 4 15d
my-dep-5f8dfc8c78-j9fqp 1/1 Running 4 15d
nginx-deployment-7f78c49b8f-4bnjn 0/1 Pending 0 22s
nginx-deployment-7f78c49b8f-6fnq2 1/1 Running 0 7m8s
nginx-deployment-7f78c49b8f-cdqn6 0/1 Pending 0 22s
nginx-deployment-7f78c49b8f-ftzb7 1/1 Running 0 7m5s
nginx-deployment-7f78c49b8f-mjjkt 1/1 Running 0 38s
nginx-deployment-7f78c49b8f-mw5vb 1/1 Running 0 38s
nginx-deployment-7f78c49b8f-tjldz 1/1 Running 0 7m3s
nginx-deployment-7f78c49b8f-tr7v6 1/1 Running 0 38s
nginx-deployment-7f78c49b8f-ww7bw 0/1 Pending 0 22s
nginx-deployment-7f78c49b8f-z47qc 0/1 Pending 0 22s
nginx-php 1/1 Running 3 10d
web-96d5df5c8-ghb6g 1/1 Running 4 17d
web2 1/1 Running 3 10d[root@k8s-master ~]# kubectl delete hpa nginx nginx-deployment
horizontalpodautoscaler.autoscaling "nginx-deployment" deleted
[root@k8s-master ~]# kubectl get hpa
No resources found in default namespace.
[root@k8s-master ~]# kubectl top hpa
error: unknown command "hpa"
See 'kubectl top -h' for help and examples
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-dep-5f8dfc8c78-dvxp8 1/1 Running 4 15d
my-dep-5f8dfc8c78-f4ln4 1/1 Running 4 15d
my-dep-5f8dfc8c78-j9fqp 1/1 Running 4 15d
nginx-deployment-7f78c49b8f-4bnjn 0/1 Pending 0 4m4s
nginx-deployment-7f78c49b8f-6fnq2 1/1 Running 0 10m
nginx-deployment-7f78c49b8f-cdqn6 0/1 Pending 0 4m4s
nginx-deployment-7f78c49b8f-ftzb7 1/1 Running 0 10m
nginx-deployment-7f78c49b8f-mjjkt 1/1 Running 0 4m20s
nginx-deployment-7f78c49b8f-mw5vb 1/1 Running 0 4m20s
nginx-deployment-7f78c49b8f-tjldz 1/1 Running 0 10m
nginx-deployment-7f78c49b8f-tr7v6 1/1 Running 0 4m20s
nginx-deployment-7f78c49b8f-ww7bw 0/1 Pending 0 4m4s
nginx-deployment-7f78c49b8f-z47qc 0/1 Pending 0 4m4s
nginx-php 1/1 Running 3 10d
web-96d5df5c8-ghb6g 1/1 Running 4 17d
web2 1/1 Running 3 10d
[root@k8s-master ~]# kubectl scale deployment nginx-deployment --replicas=3
deployment.apps/nginx-deployment scaled
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-dep-5f8dfc8c78-dvxp8 1/1 Running 4 15d
my-dep-5f8dfc8c78-f4ln4 1/1 Running 4 15d
my-dep-5f8dfc8c78-j9fqp 1/1 Running 4 15d
nginx-deployment-7f78c49b8f-6fnq2 1/1 Running 0 11m
nginx-deployment-7f78c49b8f-ftzb7 1/1 Running 0 11m
nginx-deployment-7f78c49b8f-mjjkt 0/1 Terminating 0 5m14s
nginx-deployment-7f78c49b8f-mw5vb 0/1 Terminating 0 5m14s
nginx-deployment-7f78c49b8f-tjldz 1/1 Running 0 11m
nginx-deployment-7f78c49b8f-tr7v6 0/1 Terminating 0 5m14s
nginx-php 1/1 Running 3 10d
web-96d5df5c8-ghb6g 1/1 Running 4 17d
web2 1/1 Running 3 10d
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-dep-5f8dfc8c78-dvxp8 1/1 Running 4 15d
my-dep-5f8dfc8c78-f4ln4 1/1 Running 4 15d
my-dep-5f8dfc8c78-j9fqp 1/1 Running 4 15d
nginx-deployment-7f78c49b8f-6fnq2 1/1 Running 0 11m
nginx-deployment-7f78c49b8f-ftzb7 1/1 Running 0 11m
nginx-deployment-7f78c49b8f-mjjkt 0/1 Terminating 0 5m17s
nginx-deployment-7f78c49b8f-mw5vb 0/1 Terminating 0 5m17s
nginx-deployment-7f78c49b8f-tjldz 1/1 Running 0 11m
nginx-deployment-7f78c49b8f-tr7v6 0/1 Terminating 0 5m17s
nginx-php 1/1 Running 3 10d
web-96d5df5c8-ghb6g 1/1 Running 4 17d
web2 1/1 Running 3 10d
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-dep-5f8dfc8c78-dvxp8 1/1 Running 4 15d
my-dep-5f8dfc8c78-f4ln4 1/1 Running 4 15d
my-dep-5f8dfc8c78-j9fqp 1/1 Running 4 15d
nginx-deployment-7f78c49b8f-6fnq2 1/1 Running 0 11m
nginx-deployment-7f78c49b8f-ftzb7 1/1 Running 0 11m
nginx-deployment-7f78c49b8f-mjjkt 0/1 Terminating 0 5m21s
nginx-deployment-7f78c49b8f-mw5vb 0/1 Terminating 0 5m21s
nginx-deployment-7f78c49b8f-tjldz 1/1 Running 0 11m
nginx-deployment-7f78c49b8f-tr7v6 0/1 Terminating 0 5m21s
nginx-php 1/1 Running 3 10d
web-96d5df5c8-ghb6g 1/1 Running 4 17d
web2 1/1 Running 3 10d
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-dep-5f8dfc8c78-dvxp8 1/1 Running 4 15d
my-dep-5f8dfc8c78-f4ln4 1/1 Running 4 15d
my-dep-5f8dfc8c78-j9fqp 1/1 Running 4 15d
nginx-deployment-7f78c49b8f-6fnq2 1/1 Running 0 11m
nginx-deployment-7f78c49b8f-ftzb7 1/1 Running 0 11m
nginx-deployment-7f78c49b8f-tjldz 1/1 Running 0 11m
nginx-php 1/1 Running 3 10d
web-96d5df5c8-ghb6g 1/1 Running 4 17d
web2 1/1 Running 3 10d
回滚
kubectl rollout history deployment/web
kubectl rollout undo deployment/web # 回滚上一个版本
kubectl rollout undo deployment/web --to-revision=2 # 回滚指定版本
[root@k8s-master ~]# kubectl rollout history deployment nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
1 <none>
2 <none>
3 <none>--record参数将当前命令记录到发布记录里，方便知道回滚的镜像版本：
kubectl set image deployment nginx-deployment nginx=nginx:1.17 --record删除
kubectl delete deploy/web
kubectl delete svc/web
5. 滚动升级与回滚实现机制
滚动更新策略
每次只升级一个或多个服务，升级完成加入生产环境，不断执行这个过程，直到集群中的全部旧版升级新版本。
特点：
- 用户无感知，平滑过渡
缺点：
- 部署周期长
- 发布策略较复杂
滚动更新在K8s中实现：
- 1个Deployment
- 2个ReplicaSet
[root@k8s-master ~]# kubectl get rs
NAME DESIRED CURRENT READY AGE
my-dep-5f8dfc8c78 3 3 3 15d
nginx-deployment-644599b9c9 0 0 0 5h27m
nginx-deployment-7cf55fb7bb 0 0 0 8d
nginx-deployment-7f78c49b8f 3 3 3 5h8m
web-96d5df5c8 1 1 1 17d
ReplicaSet：副本集
- 协助Deployment做事
- Pod副本数量管理，不断对比当前Pod数量与期望Pod数量
- Deployment每次发布都会创建一个RS作为记录，用于实现回滚
kubectl get rs #查看RS记录
kubectl rollout history deployment web #版本对应RS记录
6. Pod对象：Pod设计思想、应用自修复、Init container、静态Pod
6.1 Pod对象：基本概念
- 最小部署单元
- 一组容器的集合
- 一个Pod中的容器共享网络命名空间
- Pod是短暂的
定义Pod：
apiVersion: v1
kind: Pod
metadata:name: my-pod
spec:containers:- name: container1image: nginx- name: container2image: centos
6.2 Pod对象：存在的意义
Pod为亲密性应用而存在。
亲密性应用场景：
- 两个应用之间发生文件交互
- 两个应用需要通过127.0.0.1或者socket通信（典型组合nginx+php）
- 两个应用需要发生频繁的调用
6.3 Pod对象：容器分类
- Infrastructure Container：基础容器，维护整个Pod网络空间
- InitContainers：初始化容器，先于业务容器开始执行
- Containers：业务容器，并行启动
6.4 Pod对象：应用自恢复（重启策略+健康检查）
重启策略：
- Always：当容器终止退出后，总是重启容器，默认策略。
- OnFailure：当容器异常退出（退出状态码非0）时，才重启容器。
- Never：当容器终止退出，从不重启容器。
健康检查有以下两种类型：
- livenessProbe（存活检查）：如果检查失败，将杀死容器，根据Pod的restartPolicy来操作。
- readinessProbe（就绪检查）：如果检查失败，Kubernetes会把Pod从service endpoints中剔除。
持久运行的程序，例如nginx、MySQL、jar
计划任务 cronjob
数据处理程序 job
if [ 1 == 2 ]; then
echo yes
exit 1
else
exit 0
fi
1、tcp端口探测
2、curl http://ip
3、ping检查服务器状态
4、端口是监听
支持以下三种检查方法：
- httpGet：发送HTTP请求，返回200-400范围状态码为成功。
- exec：执行Shell命令返回状态码是0为成功。
- tcpSocket：发起TCP Socket建立成功。
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: webname: web
spec:replicas: 1selector:matchLabels:app: webtemplate:metadata:labels:app: webspec:restartPolicy: Alwayscontainers:- image: lizhenliang/java-demoname: java-demolivenessProbe:tcpSocket:port: 8080initialDelaySeconds: 30periodSeconds: 20
参考文档: https://kubernetes.io/docs/tasks/configure-pod-container/configureliveness-readiness-probes/
6.5 Pod对象：Init container
Init container：
- 基本支持所有普通容器特征
- 优先普通容器执行
应用场景：
- 控制普通容器启动，初始容器完成后才会启动业务容器
- 初始化配置，例如下载应用配置文件、注册信息等
apiVersion: v1
kind: Pod
metadata:
name: init-demo
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: workdir
mountPath: /usr/share/nginx/html
initContainers:
- name: install
image: busybox
command:
- wget
- "-O"
- "/work-dir/index.html"
- http://kubernetes.io
volumeMounts:
- name: workdir
mountPath: "/work-dir"
dnsPolicy: Default
volumes:
- name: workdir
emptyDir: {}
6.6 Pod对象：静态Pod
静态Pod特点：
- Pod由特定节点上的kubelet管理
- 不能使用控制器
- Pod名称标识当前节点名称
在kubelet配置文件启用静态Pod：
vi /var/lib/kubelet/config.yaml
...
staticPodPath: /etc/kubernetes/manifests
...
将部署的pod yaml放到该目录会由kubelet自动创建
课后作业
1、在节点上配置kubelet托管启动一个pod
- 节点：k8s-node1
- pod名称：web
- 镜像：nginx
# 检查是否启用
cat /var/lib/kubelet/config.yaml
staticPodPath: /etc/kubernetes/manifestsvi /etc/kubernetes/manifests/web.yaml
apiVersion: v1
kind: Pod
metadata:name: webnamespace:labels:app: myapp
spec:containers:- name: nginximage: nginx
# 注：如果不生效，检查配置文件是否启用该功能
2、向pod中添加一个init容器，init容器创建一个空文件，如果该空文件没有被检测到pod退出
- pod名称：web
apiVersion: v1
kind: Pod
metadata:name: web
spec:initContainers:- name: initimage: nginxcommand:- touch- /opt/testvolumeMounts:- name: datamountPath: /optcontainers:- name: nginximage: nginxvolumeMounts:- name: datamountPath: /optlivenessProbe:exec:command:- cat- /opt/test
restartPolicy: Never
volumes:
- name: dataemptyDir: {}
3、创建一个deployment 副本数 3，然后滚动更新镜像版本，并记录这个更新记录，最后再回滚到上一个版本
- 名称：nginx
- 镜像版本：1.16
- 更新镜像版本：1.17
kubectl create deployment web --image=nginx:1.16
kubectl set image deployment web nginx=nginx:1.17 --record
kubectl rollout history deploy web # 查看版本记录
kubectl rollout undo deployment web # 回滚到上一个版本
kubectl rollout undo deployment web --to-revision=1 # 也可以回滚到指定版本
4、给web deployment扩容副本数为3
kubectl scale deployment web --replicas=3
5、创建一个pod，其中运行着nginx、redis、memcached、consul 4个容器
apiVersion: v1
kind: Pod
metadata:name: myapp-podlabels:app: myapp
spec:containers:- name: nginximage: nginx- name: redisimage: redis- name: memcacheimage: memcached- name: consulimage: consul
6、把deployment输出json文件，再删除创建的deployment
kubectl get deployments.apps web -o json > web.json
kubectl delete deployments.apps web
7、生成一个deployment yaml文件保存到/opt/deploy.yaml
- 名称：web
- 标签：app_env_stage=dev
kubectl create deployment web --image=nginx --dry-run=client -o yaml > /opt/deploy.yaml# 再修改标签
apiVersion: apps/v1
kind: Deployment
metadata:name: java-demo
spec:replicas: 3selector:matchLabels:app_env_stage=devtemplate:metadata:labels:app_env_stage=devspec:containers:- name: nginximage: nginx
Kubernetes CKA认证运维工程师笔记-Kubernetes应用程序生命周期管理相关推荐
- Kubernetes CKA认证运维工程师笔记-Kubernetes安全
Kubernetes CKA认证运维工程师笔记-Kubernetes安全 1. Kubernetes安全框架 2. 鉴权,授权,准入控制 2.1 鉴权 2.2 授权 2.3 准入控制 3. 基于角色的 ...
- Kubernetes CKA认证运维工程师笔记-Kubernetes网络
Kubernetes CKA认证运维工程师笔记-Kubernetes网络 1. Service 存在的意义 2. Pod与Service的关系 3. Service三种常用类型 4. Service代 ...
- Kubernetes CKA认证运维工程师笔记-Kubernetes调度
Kubernetes CKA认证运维工程师笔记-Kubernetes应用程序生命周期管理 1. 创建一个Pod的工作流程 2. Pod中影响调度的主要属性 3. 资源限制对Pod调度的影响 4. no ...
- Kubernetes CKA认证运维工程师笔记-Kubernetes监控与日志
Kubernetes CKA认证运维工程师笔记-Kubernetes监控与日志 1. 查看集群资源状况 2. 监控集群资源利用率 3. 管理K8s组件日志 4. 管理K8s应用日志 1. 查看集群资源 ...
- Kubernetes CKA认证运维工程师笔记-Docker快速入门
Kubernetes CKA认证运维工程师笔记-Docker快速入门 1. Docker 概念与安装 1.1 Docker 是什么 1.2 Docker 基本组成 1.3 版本与支持平台 1.4 Do ...
- SRE运维工程师笔记-Linux基础入门
SRE运维工程师笔记-Linux基础入门 1. Linux基础 1.1 用户类型 1.2 终端terminal 1.2.1 终端类型 1.2.2 查看当前的终端设备 1.3 交互式接口 1.3.1 交 ...
- SRE运维工程师笔记-Linux用户组和权限管理
SRE运维工程师笔记-Linux用户组和权限管理 用户.组和权限 内容概述 1. Linux安全模型 1.1 用户 1.2 用户组 1.3 用户和组的关系 1.4 安全上下文 2. 用户和组的配置文件 ...
- SRE运维工程师笔记-文件查找和压缩
SRE运维工程师笔记-文件查找和压缩 1. 文件查找 1.1 locate 1.2 find 1.2.1 指定搜索目录层级 1.2.2 对每个目录先处理目录内的文件,再处理目录本身 1.2.3 根据文 ...
- SRE运维工程师笔记-Linux文件管理和IO重定向
SRE运维工程师笔记-Linux文件管理和IO重定向 1. 文件系统目录结构 1.1 文件系统的目录结构 1.2 常见的文件系统目录功能 1.3 应用程序的组成部分 1.4 CentOS 7 以后版本 ...
最新文章
- sqlmap 常用操作
- html5父子页面数据传递,使用iframe标签嵌套页面时 如何进行父子页面通讯/传值...
- 一种关于感受野尺寸计算的思路
- 排序_简单排序_选择排序
- Monkey Server自动化脚本
- 组态王6.55连接MySql数据库（笔记）
- 使用Mockito在Java中进行模拟入门
- 少儿编程100讲轻松学python（二）-python cv2模块怎么安装
- dio设置自定义post请求_Flutter Dio简单二次封装和自定义Header
- 将枚举的键值绑定到下拉列表框
- localhost访问容器mysql失败_Docker - Spring Boot应用程序 - 无法访问localhost上的MySql服务器...
- PIE Engine系列1 遥感数据下载器的实现（含源码）
- 解决linux共享文件夹丢失的问题
- 什么是谷歌趋势（Google Trends）
- K-armed Bandit
- Keras实现小数量集图片分类——6类别Birds数据集分类
- 利用matlab将三维数据画成三维立体图
- 谢文的yiqi有戏没？
- linux编辑搜索命令,Linux 命令大全提供 500 多个 Linux 命令搜索
- 黑马程序员---IT行业调查报告