k8s集群的搭建-云服务器

服务器准备

主机名	公网IP	内网IP	系统	配置
k8s-master	119.3.168.188	192.168.0.194	CentOS 7.6	4核 16G
k8s-node1	121.36.55.3	192.168.0.130	CentOS 7.6	4核 16G
k8s-node2	124.70.19.106	192.168.0.130	CentOS 7.6	4核 16G

设置主机名

hostnamectl set-hostname k8s-master
hostnamectl set-hostname k8s-node1
hostnamectl set-hostname k8s-node2

配置hosts文件

cat >> /etc/hosts<<EOF
192.168.0.194            k8s-master
192.168.0.130            k8s-node1
192.168.0.245             k8s-node2
EOF

安装docker

卸载旧版本（未安装过则跳过）

yum remove docker docker-common container-selinux docker-selinux docker-engine

安装新版本

wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo# 安装必要工具集
yum install -y yum-utils# 添加docker的yum源
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo #阿里云# 更新yum缓存
yum makecache fast# 查看docker版本信息
yum list docker-ce --showduplicates | sort -r# 挑选指定版本安装 yum -y install docker-ce-<版本号>
yum -y install docker-ce-20.10.11-3.el7# 启动docker并设置开机自启
systemctl enable docker && systemctl start docker# 检查docker版本
docker -v

配置daemon.json文件

cat >/etc/docker/daemon.json <<EOF
{"exec-opts": ["native.cgroupdriver=systemd"],"log-driver": "json-file","log-opts":{"max-size": "100m"},"registry-mirrors": ["https://82m9ar63.mirror.aliyuncs.com"]
}
EOF# 重启docker
systemctl daemon-reload
systemctl enable docker && systemctl restart docker && systemctl status docker

安装kubeadm（三台）

环境配置

# 安装一些依赖包
yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git# 将 SELinux 设置为 permissive 模式（相当于将其禁用）
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config# 关闭swap
swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab# 关闭防火墙，设置 iptables 检查桥接流量
systemctl stop firewalld && systemctl disable firewalld
yum -y install iptables-services && systemctl start iptables && systemctl enable iptables && iptables -F && service iptables save
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOFcat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

安装kubelet、kubeadm、kubectl

# 配置阿里源
cat  > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF# 安装 kubelet kubeadm kubectl
yum install -y kubelet-1.20.11 kubectl-1.20.11 kubeadm-1.20.11# systemctl在enable、disable、mask子命令里面增加了--now选项，可以激活同时启动服务，激活同时停止服务等
systemctl enable --now kubelet# 查看安装的版本
kubelet --version

如果想卸载k8s组件的话可以进行下面命令：

# 卸载K8s组件前，先执行kubeadm reset命令，清空K8s集群设置
echo y|kubeadm reset# 卸载管理组件
yum erase -y kubelet kubectl kubeadm kubernetes-cni

下载必须镜像（三台）

本来直接用kubeadm init就行，但是由于init命令是从k8s.gcr.io网站上下载镜像，被墙了，所以需要写个脚本把这些镜像下好

kubeadm init主要执行操作

[init]：指定版本进行初始化操作
[preflight] ：初始化前的检查和下载所需要的Docker镜像文件
[kubelet-start] ：生成kubelet的配置文件”/var/lib/kubelet/config.yaml”，没有这个文件kubelet无法启动，所以初始化之前的kubelet实际上启动失败。
[certificates]：生成Kubernetes使用的证书，存放在/etc/kubernetes/pki目录中。
[kubeconfig] ：生成 KubeConfig 文件，存放在/etc/kubernetes目录中，组件之间通信需要使用对应文件。
[control-plane]：使用/etc/kubernetes/manifest目录下的YAML文件，安装 Master 组件。
[etcd]：使用/etc/kubernetes/manifest/etcd.yaml安装Etcd服务。
[wait-control-plane]：等待control-plan部署的Master组件启动。
[apiclient]：检查Master组件服务状态。
[uploadconfig]：更新配置
[kubelet]：使用configMap配置kubelet。
[patchnode]：更新CNI信息到Node上，通过注释的方式记录。
[mark-control-plane]：为当前节点打标签，打了角色Master，和不可调度标签，这样默认就不会使用Master节点来运行Pod。
[bootstrap-token]：生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到
[addons]：安装附加组件CoreDNS和kube-proxy

查看需要下载的镜像

kubeadm config images list# 输出结果, 这些都是K8S的必要组件, 但是由于被墙, 是不能直接docker pull下来的
k8s.gcr.io/kube-apiserver:v1.20.15
k8s.gcr.io/kube-controller-manager:v1.20.15
k8s.gcr.io/kube-scheduler:v1.20.15
k8s.gcr.io/kube-proxy:v1.20.15
k8s.gcr.io/pause:3.2
k8s.gcr.io/etcd:3.4.13-0
k8s.gcr.io/coredns:1.7.0

编写pull脚本

## 位置自己确定，记住就行
cat >/root/k8s-script/pull_k8s_images.sh << "EOF"
# 内容为
set -o errexit
set -o nounset
set -o pipefail##这里定义需要下载的版本
KUBE_VERSION=v1.20.15
KUBE_PAUSE_VERSION=3.2
ETCD_VERSION=3.4.13-0
DNS_VERSION=1.7.0##这是原来被墙的仓库
GCR_URL=k8s.gcr.io##这里就是写你要使用的仓库,也可以使用gotok8s
DOCKERHUB_URL=registry.cn-hangzhou.aliyuncs.com/google_containers##这里是镜像列表
images=(
kube-proxy:${KUBE_VERSION}
kube-scheduler:${KUBE_VERSION}
kube-controller-manager:${KUBE_VERSION}
kube-apiserver:${KUBE_VERSION}
pause:${KUBE_PAUSE_VERSION}
etcd:${ETCD_VERSION}
coredns:${DNS_VERSION}
)## 这里是拉取和改名的循环语句, 先下载, 再tag重命名生成需要的镜像, 再删除下载的镜像
for imageName in ${images[@]} ; dodocker pull $DOCKERHUB_URL/$imageNamedocker tag $DOCKERHUB_URL/$imageName $GCR_URL/$imageNamedocker rmi $DOCKERHUB_URL/$imageName
done
EOF

推送脚本到node节点中

# 示例
scp /root/k8s-script/pull_k8s_images.sh root@IP地址:/root/k8s-script/scp /root/k8s-script/pull_k8s_images.sh root@121.36.55.3:/root/k8s-script/pull_k8s_images.sh
scp /root/k8s-script/pull_k8s_images.sh root@124.70.19.106:/root/k8s-script/pull_k8s_images.sh

执行脚本

bash /root/k8s-script/pull_k8s_images.sh

查看下载结果

docker images
REPOSITORY                           TAG        IMAGE ID       CREATED         SIZE
k8s.gcr.io/kube-proxy                v1.20.15   46e2cd1b2594   4 months ago    99.7MB
k8s.gcr.io/kube-scheduler            v1.20.15   9155e4deabb3   4 months ago    47.3MB
k8s.gcr.io/kube-controller-manager   v1.20.15   d6296d0e06d2   4 months ago    116MB
k8s.gcr.io/kube-apiserver            v1.20.15   323f6347f5e2   4 months ago    122MB
k8s.gcr.io/etcd                      3.4.13-0   0369cf4303ff   21 months ago   253MB
k8s.gcr.io/coredns                   1.7.0      bfe3a36ebd25   23 months ago   45.2MB
k8s.gcr.io/pause                     3.2        80d28bedfe5d   2 years ago     683kB

初始化主节点（只有主节点）

kubeadm-config.yaml

# 修改项下面标出
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:- system:bootstrappers:kubeadm:default-node-tokentoken: abcdef.0123456789abcdefttl: 24h0m0susages:- signing- authentication
kind: InitConfiguration
localAPIEndpoint:advertiseAddress: 10.0.1.43     # 本机IPbindPort: 6443
nodeRegistration:criSocket: /var/run/dockershim.sockname: k8s-master         # 本主机名taints:- effect: NoSchedulekey: node-role.kubernetes.io/master
---
apiServer:timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}           # 虚拟IP和haproxy端口（可以不填写）
dns:type: CoreDNS
etcd:local:dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers     # 镜像仓库源要根据自己实际情况修改
kind: ClusterConfiguration
kubernetesVersion: v1.20.15      # 修改版本, 与前面版本一致, 也可通过 kubeadm version 查看版本
networking:dnsDomain: cluster.localpodSubnet: "10.244.0.0/16"    # 新增pod子网, 固定该IP即可serviceSubnet: 10.96.0.0/12
scheduler: {}# 新增下面设置, 固定即可
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

kubeadm init --config=kubeadm-config.yaml | tee kubeadm-init.log# 正常运行结果
......
Your Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/configAlternatively, if you are the root user, you can run:export KUBECONFIG=/etc/kubernetes/admin.confYou should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:https://kubernetes.io/docs/concepts/cluster-administration/addons/Then you can join any number of worker nodes by running the following on each as root:
......

根据提示操作

# 在master上运行
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config# 推送node{1..X}机器上，如果/root/.kube/config没有目录要手动创建
scp /etc/kubernetes/admin.conf root@121.36.55.3:/root/.kube/config
scp /etc/kubernetes/admin.conf root@124.70.19.106:/root/.kube/config

查看当前节点状态

[root@k8s-master ~]# kubectl get nodes
NAME         STATUS     ROLES                  AGE     VERSION
k8s-master   NotReady   control-plane,master   9m27s   v1.20.11

将子节点加入到主节点下面（在子节点上操作）

还是在主节点的init命令的输出日志下, 有子节点的加入命令, 在两台子节点服务器上运行

kubeadm join MasterIP地址:6443 --token xxxxxx \--discovery-token-ca-cert-hash sha256:xxxxxx #正常运行结果
[preflight] Running pre-flight checks[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.7. Latest validated version: 19.03
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

查看集群节点

kubectl get nodes[root@k8s-node2 ~]# kubectl get nodes
NAME         STATUS     ROLES                  AGE     VERSION
k8s-master   NotReady   control-plane,master   14m     v1.20.11
k8s-node1    NotReady   <none>                 3m39s   v1.20.11
k8s-node2    NotReady   <none>                 57s     v1.20.11

部署flannel网络（主节点操作）

安装flannel网络插件

# 先拉取镜像，此过程国内速度比较慢
docker pull quay.io/coreos/flannel:v0.14.0

配置flannel

# 去https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml弄一个yml文件
kubectl create -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created# 查看pod, 可以看到flannel组件已经运行起来了. 默认系统组件都安装在 kube-system 这个命名空间(namespace)下
[root@k8s-master ~]# kubectl get pod -n kube-system
NAME                                 READY   STATUS    RESTARTS   AGE
coredns-7f89b7bc75-6wmct             1/1     Running   0          51m
coredns-7f89b7bc75-nvnnr             1/1     Running   0          51m
etcd-k8s-master                      1/1     Running   0          51m
kube-apiserver-k8s-master            1/1     Running   0          51m
kube-controller-manager-k8s-master   1/1     Running   0          51m
kube-flannel-ds-dbwqc                1/1     Running   0          12m
kube-flannel-ds-pfk6t                1/1     Running   0          12m
kube-flannel-ds-q8tkd                1/1     Running   0          12m
kube-proxy-jcll5                     1/1     Running   0          40m
kube-proxy-l68cn                     1/1     Running   0          37m
kube-proxy-qwf5z                     1/1     Running   0          51m
kube-scheduler-k8s-master            1/1     Running   0          51m# 再次查看node, 发现状态已经变成了 Ready
[root@k8s-master ~]# kubectl get node
NAME         STATUS   ROLES                  AGE   VERSION
k8s-master   Ready    control-plane,master   51m   v1.20.11
k8s-node1    Ready    <none>                 40m   v1.20.11
k8s-node2    Ready    <none>                 37m   v1.20.11

如果想要卸载flannel则运行下面命令：

kubectl delete -f kube-flannel.yml

部署bashboard

下载dashboard配置文件

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml

vim recommended.yaml #修改两处

kubectl apply -f recommended.yaml

使用命令查看哪个镜像拉取失败了手动拉取镜像即可！

kubectl describe pod dashboard-metrics-scraper-64bcc67c9c-g55jh -n kubernetes-dashboard

**默认，**基于安全原因，集群并不会在control-plane节点上部署Pods。如果你需要在control-plane上部署Pods，比如用于development的单主机集群，需要执行如下：

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

编辑dashboard-admin.yaml文件

apiVersion: v1
kind: ServiceAccount
metadata:labels:k8s-app: kubernetes-dashboardname: dashboard-adminnamespace: kubernetes-dashboard

kubectl apply -f dashboard-admin.yaml

编辑dashboard-admin-bind-cluster-role.yaml文件

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: dashboard-admin-bind-cluster-rolelabels:k8s-app: kubernetes-dashboard
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: cluster-admin
subjects:
- kind: ServiceAccountname: dashboard-adminnamespace: kubernetes-dashboard

kubectl apply -f dashboard-admin-bind-cluster-role.yaml

查获pod 部署详情

kubectl -n kubernetes-dashboard describe pod  dashboard-metrics-scraper-64bcc67c9c-g55jh
kubectl -n kubernetes-dashboard describe pod kubernetes-dashboard-5c8bd6b59-tpjhw

创建访问账户，获取token

获取Dashboard Token

kubectl -n kubernetes-dashboard create token dashboard-admin
或者
# 创建账号
kubectl create serviceaccount dashboard-admin -n kubernetes-dashboard# 授权
kubectl create clusterrolebinding dashboard-admin-rb --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:dashboard-admin

获取token：

kubectl get secrets -n kubernetes-dashboard | grep dashboard-admin
kubectl describe secrets dashboard-admin-token-546rb -n kubernetes-dashboard

Horizontal Pod Autoscaler(HPA)

可以实现通过手工执行kubectl scale命令实现Pod扩容或缩容，但是这显然不符合Kubernetes的定位目标–自动化、智能化。 Kubernetes期望可以实现通过监测Pod的使用情况，实现pod数量的自动调整，于是就产生了Horizontal Pod Autoscaler（HPA）这种控制器。

HPA可以获取每个Pod利用率，然后和HPA中定义的指标进行对比，同时计算出需要伸缩的具体值，最后实现Pod的数量的调整。其实HPA与之前的Deployment一样，也属于一种Kubernetes资源对象，它通过追踪分析RC控制的所有目标Pod的负载变化情况，来确定是否需要针对性地调整目标Pod的副本数，这是HPA的实现原理。

metrics-server可以用来收集集群中的资源使用情况

# 安装git
[root@k8s-master01 ~]# yum install git -y
# 获取metrics-server, 注意使用的版本
[root@k8s-master01 ~]# git clone -b v0.3.6 https://github.com/kubernetes-incubator/metrics-server
# 修改deployment, 注意修改的是镜像和初始化参数
[root@k8s-master01 ~]# cd /root/metrics-server/deploy/1.8+/
[root@k8s-master01 1.8+]# vim metrics-server-deployment.yaml
按图中添加下面选项
hostNetwork: true
image: registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6
args:
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP

# 安装metrics-server
[root@k8s-master01 1.8+]# kubectl apply -f ./# 查看pod运行情况
[root@k8s-master01 1.8+]# kubectl get pod -n kube-system
metrics-server-6b976979db-2xwbj   1/1     Running   0          90s# 使用kubectl top node 查看资源使用情况
[root@k8s-master01 1.8+]# kubectl top node
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
k8s-master01   289m         14%    1582Mi          54%
k8s-node01     81m          4%     1195Mi          40%
k8s-node02     72m          3%     1211Mi          41%
[root@k8s-master01 1.8+]# kubectl top pod -n kube-system
NAME                              CPU(cores)   MEMORY(bytes)
coredns-6955765f44-7ptsb          3m           9Mi
coredns-6955765f44-vcwr5          3m           8Mi
etcd-master                       14m          145Mi
...
# 至此,metrics-server安装完成

准备deployment和servie

创建pc-hpa-pod.yaml文件，内容如下：

apiVersion: apps/v1
kind: Deployment
metadata:name: nginxnamespace: dev
spec:strategy: # 策略type: RollingUpdate # 滚动更新策略replicas: 1selector:matchLabels:app: nginx-podtemplate:metadata:labels:app: nginx-podspec:containers:- name: nginximage: nginx:1.17.1resources: # 资源配额limits:  # 限制资源（上限）cpu: "1" # CPU限制，单位是core数requests: # 请求资源（下限）cpu: "100m"  # CPU限制，单位是core数

# 创建deployment
[root@k8s-master01 1.8+]# kubectl run nginx --image=nginx:1.17.1 --requests=cpu=100m -n dev
# 创建service
[root@k8s-master01 1.8+]# kubectl expose deployment nginx --type=NodePort --port=80 -n dev

# 查看
[root@k8s-master01 1.8+]# kubectl get deployment,pod,svc -n dev
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1           47sNAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-7df9756ccc-bh8dr   1/1     Running   0          47sNAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/nginx   NodePort   10.101.18.29   <none>        80:31830/TCP   35s

部署HPA

创建pc-hpa.yaml文件，内容如下：

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:name: pc-hpanamespace: dev
spec:minReplicas: 1  #最小pod数量maxReplicas: 10 #最大pod数量targetCPUUtilizationPercentage: 3 # CPU使用率指标scaleTargetRef:   # 指定要控制的nginx信息apiVersion:  /v1kind: Deploymentname: nginx

# 创建hpa
[root@k8s-master01 1.8+]# kubectl create -f pc-hpa.yaml
horizontalpodautoscaler.autoscaling/pc-hpa created# 查看hpa[root@k8s-master01 1.8+]# kubectl get hpa -n dev
NAME     REFERENCE          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
pc-hpa   Deployment/nginx   0%/3%     1         10        1          62s

测试

使用压测工具对service地址192.168.5.4:31830进行压测，然后通过控制台查看hpa和pod的变化

hpa变化

[root@k8s-master01 ~]# kubectl get hpa -n dev -w
NAME   REFERENCE      TARGETS  MINPODS  MAXPODS  REPLICAS  AGE
pc-hpa  Deployment/nginx  0%/3%   1     10     1      4m11s
pc-hpa  Deployment/nginx  0%/3%   1     10     1      5m19s
pc-hpa  Deployment/nginx  22%/3%   1     10     1      6m50s
pc-hpa  Deployment/nginx  22%/3%   1     10     4      7m5s
pc-hpa  Deployment/nginx  22%/3%   1     10     8      7m21s
pc-hpa  Deployment/nginx  6%/3%   1     10     8      7m51s
pc-hpa  Deployment/nginx  0%/3%   1     10     8      9m6s
pc-hpa  Deployment/nginx  0%/3%   1     10     8      13m
pc-hpa  Deployment/nginx  0%/3%   1     10     1      14m

deployment变化

[root@k8s-master01 ~]# kubectl get deployment -n dev -w
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   1/1     1            1           11m
nginx   1/4     1            1           13m
nginx   1/4     1            1           13m
nginx   1/4     1            1           13m
nginx   1/4     4            1           13m
nginx   1/8     4            1           14m
nginx   1/8     4            1           14m
nginx   1/8     4            1           14m
nginx   1/8     8            1           14m
nginx   2/8     8            2           14m
nginx   3/8     8            3           14m
nginx   4/8     8            4           14m
nginx   5/8     8            5           14m
nginx   6/8     8            6           14m
nginx   7/8     8            7           14m
nginx   8/8     8            8           15m
nginx   8/1     8            8           20m
nginx   8/1     8            8           20m
nginx   1/1     1            1           20m

pod变化

[root@k8s-master01 ~]# kubectl get pods -n dev -w
NAME                     READY   STATUS    RESTARTS   AGE
nginx-7df9756ccc-bh8dr   1/1     Running   0          11m
nginx-7df9756ccc-cpgrv   0/1     Pending   0          0s
nginx-7df9756ccc-8zhwk   0/1     Pending   0          0s
nginx-7df9756ccc-rr9bn   0/1     Pending   0          0s
nginx-7df9756ccc-cpgrv   0/1     ContainerCreating   0          0s
nginx-7df9756ccc-8zhwk   0/1     ContainerCreating   0          0s
nginx-7df9756ccc-rr9bn   0/1     ContainerCreating   0          0s
nginx-7df9756ccc-m9gsj   0/1     Pending             0          0s
nginx-7df9756ccc-g56qb   0/1     Pending             0          0s
nginx-7df9756ccc-sl9c6   0/1     Pending             0          0s
nginx-7df9756ccc-fgst7   0/1     Pending             0          0s
nginx-7df9756ccc-g56qb   0/1     ContainerCreating   0          0s
nginx-7df9756ccc-m9gsj   0/1     ContainerCreating   0          0s
nginx-7df9756ccc-sl9c6   0/1     ContainerCreating   0          0s
nginx-7df9756ccc-fgst7   0/1     ContainerCreating   0          0s
nginx-7df9756ccc-8zhwk   1/1     Running             0          19s
nginx-7df9756ccc-rr9bn   1/1     Running             0          30s
nginx-7df9756ccc-m9gsj   1/1     Running             0          21s
nginx-7df9756ccc-cpgrv   1/1     Running             0          47s
nginx-7df9756ccc-sl9c6   1/1     Running             0          33s
nginx-7df9756ccc-g56qb   1/1     Running             0          48s
nginx-7df9756ccc-fgst7   1/1     Running             0          66s
nginx-7df9756ccc-fgst7   1/1     Terminating         0          6m50s
nginx-7df9756ccc-8zhwk   1/1     Terminating         0          7m5s
nginx-7df9756ccc-cpgrv   1/1     Terminating         0          7m5s
nginx-7df9756ccc-g56qb   1/1     Terminating         0          6m50s
nginx-7df9756ccc-rr9bn   1/1     Terminating         0          7m5s
nginx-7df9756ccc-m9gsj   1/1     Terminating         0          6m50s
nginx-7df9756ccc-sl9c6   1/1     Terminating         0          6m50s

参考文章

kubeadm安装k8s集群（阿里云服务）
kubeadm安装k8s集群