CentOS7 安装部署k8s

1、官网说明：

Kubernetes 安装 kubeadm
使用kubeadm创建Kubernetes集群

2、准备工作

学习和练手，一切从简！

直接使用 root 身份
关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
setenforce 0

sed -i ‘s/^SELINUX=enforcing$/SELINUX=disabled/’ /etc/selinux/config && setenforce 0

或者 vim /etc/selinux/config 修改

关闭 swap
swapoff -a

vim /etc/fstab

 ## /etc/fstab# Created by anaconda on Mon Jun 28 23:11:04 2021## Accessible filesystems, by reference, are maintained under '/dev/disk'# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info#/dev/mapper/cl-root     /                       xfs     defaults        0 0UUID=0b4346b6-cee1-4abb-932e-0c1cb4cda404 /boot                   xfs     defaults        0 0/dev/mapper/cl-home     /home                   xfs     defaults        0 0# wzh 20211026 for k8s# /dev/mapper/cl-swap     swap                    swap    defaults        0 0

修改并加上所有节点主机名
vim /etc/hosts

 127.0.0.1 centos7-141192.168.0.141 centos7-141192.168.0.142 centos7-142192.168.0.143 centos7-143192.168.0.144 centos7-144

验证
free -m

               total        used        free      shared  buff/cache   availableMem:           3789         193        2961           8         634        3350Swap:             0           0           0

3、安装 Docker

官方文档
Install Docker Engine on CentOS

简单摘录一下步骤：

yum install -y yum-utils
yum-config-manager
–add-repo
https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce docker-ce-cli containerd.io

设置 Docker 镜像，并设置cgroupDriver
vim /etc/docker/daemon.json

 {"exec-opts":["native.cgroupdriver=systemd"],"registry-mirrors": ["https://2vgbfb0x.mirror.aliyuncs.com"]}

启动服务，并设置开机启动
systemctl enable docker && systemctl start docker
验证 Docker
docker run hello-world

4、安装kubectl、kubelet和kubeadm

配置yum源

 cat <<EOF > /etc/yum.repos.d/kubernetes.repo[kubernetes]name=Kubernetesbaseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64enabled=1gpgcheck=1repo_gpgcheck=1gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpghttps://packages.cloud.google.com/yum/doc/rpm-package-key.gpgEOF

yum install -y kubelet kubeadm kubectl

systemctl enable kubelet && systemctl start kubelet

5、master 节点执行初始化

配置初始化文件
mkdir working && cd working

kubeadm config print init-defaults > kubeadm-config.yaml

vim kubeadm-config.yaml

修改
1). advertiseAddress: 192.168.0.141
2). imageRepository: registry.aliyuncs.com/google_containers
3). name: 改成 /etc/hosts 中设置好的主机名称

 apiVersion: kubeadm.k8s.io/v1beta3bootstrapTokens:- groups:- system:bootstrappers:kubeadm:default-node-tokentoken: abcdef.0123456789abcdefttl: 24h0m0susages:- signing- authenticationkind: InitConfigurationlocalAPIEndpoint:advertiseAddress: 192.168.0.141bindPort: 6443nodeRegistration:criSocket: /var/run/dockershim.sockimagePullPolicy: IfNotPresentname: centos7-141taints: null---apiServer:timeoutForControlPlane: 4m0sapiVersion: kubeadm.k8s.io/v1beta3certificatesDir: /etc/kubernetes/pkiclusterName: kubernetescontrollerManager: {}dns: {}etcd:local:dataDir: /var/lib/etcdimageRepository: registry.aliyuncs.com/google_containerskind: ClusterConfigurationkubernetesVersion: 1.22.0networking:podSubnet: 10.244.0.0/16dnsDomain: cluster.local

预先拉取所需镜像
kubeadm config images pull --config=kubeadm-config.yaml

  [config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.22.0[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.22.0[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.22.0[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.22.0[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.5[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.0-0[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.8.4

这一步非必需，预先拉取可以提前发现失败的 images，提前修改为镜像方式获取
只需要执行一次，可以 docker images 确认一下

初始化
加上 tee kubeadm-init.log，方便后续查看 token 和初始化信息
kubeadm init --config=kubeadm-config.yaml | tee kubeadm-init.log

 [init] Using Kubernetes version: v1.22.0[preflight] Running pre-flight checks[preflight] Pulling images required for setting up a Kubernetes cluster[preflight] This might take a minute or two, depending on the speed of your internet connection[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'...Your Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/configAlternatively, if you are the root user, you can run:export KUBECONFIG=/etc/kubernetes/admin.confYou should now deploy a pod network to the cluster.Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:https://kubernetes.io/docs/concepts/cluster-administration/addons/Then you can join any number of worker nodes by running the following on each as root:kubeadm join 192.168.0.141:6443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:57df376d612009f381bd3f3835464578666536080c6f779cffcf8bc90af10930

按照提示，root 身份简单设置

echo “export KUBECONFIG=/etc/kubernetes/admin.conf” >> /etc/profile

启动生效
#source /etc/profile

大约1分钟后，确认所有服务健康状态：Healthy
kubectl get cs

 Warning: v1 ComponentStatus is deprecated in v1.19+NAME                 STATUS      MESSAGE                                                                                       ERRORscheduler            Unhealthy   Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused   controller-manager   Healthy     ok                                                                                            etcd-0               Healthy     {"health":"true","reason":""}

我这里scheduler总是Unhealthy，手工修改以下 2 个文件

vim /etc/kubernetes/manifests/kube-scheduler.yaml
vim /etc/kubernetes/manifests/kube-controller-manager.yaml

删除或者注释掉 - --port=0

重启kubelet服务生效
systemctl restart kubelet

再等1分钟
kubectl get cs

 Warning: v1 ComponentStatus is deprecated in v1.19+NAME                 STATUS    MESSAGE                         ERRORscheduler            Healthy   ok                              etcd-0               Healthy   {"health":"true","reason":""}   controller-manager   Healthy   ok

如果发生错误，随时 kubeadm reset 再重来

确认 configmap 配置状态
kubectl get -n kube-system configmap

 NAME                                 DATA   AGEcoredns                              1      9m54sextension-apiserver-authentication   6      10mkube-flannel-cfg                     2      43skube-proxy                           2      9m54skube-root-ca.crt                     1      9m43skubeadm-config                       1      9m56skubelet-config-1.22                  1      9m56s

6、master节点安装pod网络

获取 kube-flannel.yml

 curl  -o  kube-flannel.yml  https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

把yml文件中的所有的quay.io改为quay.mirrors.ustc.edu.cn

sed  -i  's/quay.io/quay.mirrors.ustc.edu.cn/g'   kube-flannel.yml

或者

sed  -i  's/quay.io/quay-mirror.qiniu.com/g'   kube-flannel.yml

生成 flannel 插件pod
kubectl apply -f kube-flannel.yml

 Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+podsecuritypolicy.policy/psp.flannel.unprivileged createdclusterrole.rbac.authorization.k8s.io/flannel createdclusterrolebinding.rbac.authorization.k8s.io/flannel createdserviceaccount/flannel createdconfigmap/kube-flannel-cfg createddaemonset.apps/kube-flannel-ds created

确认配置正确
kubectl get -n kube-system configmap

 NAME                                 DATA   AGEcoredns                              1      9m54sextension-apiserver-authentication   6      10mkube-flannel-cfg                     2      43skube-proxy                           2      9m54skube-root-ca.crt                     1      9m43skubeadm-config                       1      9m56skubelet-config-1.22                  1      9m56s

确认所有的Pod都处于Running状态
kubectl get pod -n kube-system

 NAME                                  READY   STATUS    RESTARTS      AGEcoredns-7f6cbbb7b8-wb7xf              1/1     Running   0             12mcoredns-7f6cbbb7b8-ww5z4              1/1     Running   0             12metcd-centos7-141                      1/1     Running   7             12mkube-apiserver-centos7-141            1/1     Running   1             12mkube-controller-manager-centos7-141   1/1     Running   1 (12m ago)   12mkube-flannel-ds-bvvq6                 1/1     Running   0             3m31skube-proxy-8f8bq                      1/1     Running   0             12mkube-scheduler-centos7-141            1/1     Running   3 (12m ago)   12m

6、worker节点join

每一个节点服务器也和 master 主节点一样安装 Docker、kubectl、kubelet和kubeadm

如果master 重新init，则work节点join之前先执行 kubeadm reset

按照 master 初始化的输出提示加入集群

 kubeadm join 192.168.0.141:6443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:57df376d612009f381bd3f3835464578666536080c6f779cffcf8bc90af10930

返回结果大致如下

 [preflight] Running pre-flight checks[preflight] Reading configuration from the cluster...[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"[kubelet-start] Starting the kubelet[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...This node has joined the cluster:* Certificate signing request was sent to apiserver and a response was received.* The Kubelet was informed of the new secure connection details.Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

如果没有记住刚才的 token ， master 主机 # cat kubeadm-init.log 可以找到
或者 kubeadm token list

TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
abcdef.0123456789abcdef   23h         2021-11-10T08:01:53Z   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token

如果超过 24 小时没有 join ，token 过期，需要在 master 重新获取 token

kubeadm token create8mfiss.yvbnl8m319ysiflh

验证node和 Pod状态，全部为Running
kubectl get nodes

 NAME          STATUS   ROLES                  AGE     VERSIONcentos7-141   Ready    control-plane,master   30m     v1.22.2centos7-143   Ready    <none>                 7m48s   v1.22.2centos7-144   Ready    <none>                 2m22s   v1.22.2

kubectl get pods --all-namespaces

 NAMESPACE     NAME                                  READY   STATUS    RESTARTS      AGEkube-system   coredns-7f6cbbb7b8-wb7xf              1/1     Running   0             28mkube-system   coredns-7f6cbbb7b8-ww5z4              1/1     Running   0             28mkube-system   etcd-centos7-141                      1/1     Running   7             29mkube-system   kube-apiserver-centos7-141            1/1     Running   1             29mkube-system   kube-controller-manager-centos7-141   1/1     Running   1 (28m ago)   28mkube-system   kube-flannel-ds-b5sg8                 1/1     Running   0             47skube-system   kube-flannel-ds-bl9vr                 1/1     Running   0             6m13skube-system   kube-flannel-ds-bvvq6                 1/1     Running   0             19mkube-system   kube-proxy-8f8bq                      1/1     Running   0             28mkube-system   kube-proxy-j679n                      1/1     Running   0             47skube-system   kube-proxy-qczzf                      1/1     Running   0             6m13skube-system   kube-scheduler-centos7-141            1/1     Running   3 (28m ago)   28m

7、部署dashboard

dashboard官方仓库

另外写一个博文
k8s 配置dashboard

8、错误处理

[kubelet-check] The HTTP call equal to ‘curl -sSL http://localhost:10248/healthz’ failed with error: Get “http://localhost:10248/healthz”: dial tcp [::1]:10248: connect: connection refused.

给 Docker 设置 cgroupDriver
vim /etc/docker/daemon.json
```
 {"exec-opts":["native.cgroupdriver=systemd"],"registry-mirrors": ["https://2vgbfb0x.mirror.aliyuncs.com"]}
```
systemctl daemon-reload
systemctl restart docker
systemctl restart kubelet
This error is likely caused by:
- The kubelet is not running

重启kubelet 后查看kubelet status
遇到奇怪的错误: failed to run Kubelet: unable to load bootstrap kubecon…r directory
```
 ...11月 10 16:39:04 centos7-189 kubelet[14277]: E1110 16:39:04.095861   14277 server.go:294] "Failed to run kubelet" err="failed to run Kubelet: unable to load bootstrap kubecon...r directory"11月 10 16:39:04 centos7-189 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE11月 10 16:39:04 centos7-189 systemd[1]: Unit kubelet.service entered failed state.11月 10 16:39:04 centos7-189 systemd[1]: kubelet.service failed.Hint: Some lines were ellipsized, use -l to show in full.
```
因为这个电脑来回折腾，之前作为 worker node ，后来又实用普通 user 安装配置过！也许有什么遗留没有清理干净？

cat: /var/lib/kubelet/kubeadm-flags.env: 没有那个文件或目录
这个文件是 kubeadm init 生成的
所以，重新 kubeadm reset 后执行 # kubeadm init --config=kubeadm-config.yaml | tee kubeadm-init.log

重新安装kubectl、kubelet和kubeadm
他自己就恢复了？怎么回事？因为我重装了？

/var/lib/kubelet/kubeadm-flags.env 现在创建了
cat /var/lib/kubelet/kubeadm-flags.env
```
 KUBELET_KUBEADM_ARGS="--network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.5"
```
很多地方说 Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf 这个文件里面加上 --cgroup-driver=systemd，如下：
```
   Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --cgroup-driver=systemd"
```
但是，这次没有加 --cgroup-driver=systemd ，也是 ok 的！
服务器关机后再开机，发现 coredns 状态 ContainerCreating ，kube-flannel-ds-k8cgb 状态 CrashLoopBackOff ，只好kubeadm reset 后重来，发生以下错误

…
[ERROR FileContent–proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...

需要配置 ipv4 转发
vim /etc/sysctl.d/k8s.conf
```
 net.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1net.ipv4.ip_forward = 1
```
设置生效
sysctl -p /etc/sysctl.d/k8s.conf
worker 节点 join 发生 bridge-nf-call-iptables contents are not set to 1 错误
…
I1115 10:16:17.248205 14547 checks.go:432] validating if the connectivity type is via proxy or direct
[preflight] Some fatal errors occurred:
[ERROR FileContent–proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
error execution phase preflight

[root@centos7-185 ~]# kubeadm reset
之后再来设置
[root@centos7-185 ~]# echo “1” >/proc/sys/net/bridge/bridge-nf-call-iptables
确认
[root@centos7-185 ~]# cat /proc/sys/net/bridge/bridge-nf-call-iptables
1
重新 join ， ok！

9 、images 被墙时使用国内镜像地址替换

在应用yaml文件创建资源时，将文件中镜像地址进行内容替换即可：
1.k8s.gcr.io 地址替换
　　registry.cn-hangzhou.aliyuncs.com/google_containers
　　或者
　　registry.aliyuncs.com/google_containers
　　或者
　　mirrorgooglecontainers

quay.io 地址替换
quay-mirror.qiniu.com
gcr.io 地址替换
registry.aliyuncs.com