从零开始搭建一个K8S的环境

市面上有很多的K8S的书籍，但是关于如何搭建K8S的文章不多；下面笔者结合自己的经验，分享一下如何在阿里云的CentOs服务器上，搭建一个K8S的集群。

前提条件

一个阿里云的账号或者自己准备两台VM，其中一台是K8S的Master服务器，另外一台是K8S的Worker节点。
笔者自己在阿里云上面新建了2台按需付费的实例。

Master服务器：k8sMaster 172.24.137.71
Worker服务器：k8sWorker 172.24.137.72

因为Master节点和Worker节点要进行相互的网络访问,根据K8S对网络防火墙的要求，需要开通下面的端口
创建两个安全组：
Master节点加入下面的安全组

Woker节点加入下面的安全组

具体细节，请参考K8S防火墙

Master节点应该开的网络防火墙设置：

Worker节点应该开的网络防火墙设置：

1. 安装Docker

首先在Master服务器和Worker服务器上都要安装好Docker，为了加快安装的过程，需要把Docker安装文件的镜像指向国内的
阿里云镜像。如果直接用国外的，可能要1个多小时甚至更长时间下载安装包，但是如果直接指向http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.rep，快的话，分分钟的时间就下载并安装完成。

1.1 更改源为阿里源

sudo yum remove docker \docker-client \docker-client-latest \docker-common \docker-latest \docker-latest-logrotate \docker-logrotate \docker-enginesudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.rep

1.2 安装Docker

sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.rep
yum list docker-ce --showduplicates | sort -r
sudo yum makecache fast
yum install  docker-ce-18.06.3.ce-3.el7  docker-ce-cli-18.06.3.ce-3.el7  containerd.io

安装完后执行

systemctl enable docker
systemctl start docker

1.3 修改Docker的源为阿里自己的Docker Hub <可选>

安装完毕后, 建议将 docker 源替换为国内. 推荐阿里云镜像加速, 有阿里云账号即可免费使用.阿里云 -> 容器镜像服务 -> 镜像中心 -> 镜像加速

sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{"registry-mirrors": ["https://xxxxx.mirror.aliyuncs.com"],"exec-opts": ["native.cgroupdriver=systemd"],"log-driver": "json-file","log-opts": {"max-size": "100m"},"storage-driver": "overlay2"
}
EOF

其中，xxxxx自己阿里云账号的Docker Hub Register中心

安装完后执行

sudo systemctl daemon-reload
sudo systemctl restart docker

2. 安装K8S

在Master和Worker的机器上都是一样的安装步骤

2.1 修改K8S的源为阿里的源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF

2.2 安装K8S的软件

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable kubelet && systemctl start kubelet

2.3 K8S的通用网络配置

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

3. 配置Master节点和安装Dashboard

3.1 初始化Master节点

kubeadm config print init-defaults > kubeadm-init.yaml

修改模板文件的中2处配置：
将advertiseAddress: 1.2.3.4修改为本机地址
将imageRepository: k8s.gcr.io修改为imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers

[root@k8sMaster ~]# cat kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:- system:bootstrappers:kubeadm:default-node-tokentoken: abcdef.0123456789abcdefttl: 24h0m0susages:- signing- authentication
kind: InitConfiguration
localAPIEndpoint:advertiseAddress: 172.24.137.71bindPort: 6443
nodeRegistration:criSocket: /var/run/dockershim.sockname: k8smastertaints:- effect: NoSchedulekey: node-role.kubernetes.io/master
---
apiServer:timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:type: CoreDNS
etcd:local:dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.20.0
networking:dnsDomain: cluster.localserviceSubnet: 10.96.0.0/12
scheduler: {}

3.2 下载K8S Master节点的依赖的管理Docker镜像

kubeadm config images pull --config kubeadm-init.yaml

3.3 执行初始化

kubeadm init --config kubeadm-init.yaml

最后两行需要保存下来, kubeadm join …是 worker 节点加入所需要执行的命令.
其参考格式如下：

kubeadm join 172.24.137.71:6443 --token 6s1u3g.rns2x03czqjux1x1 \--discovery-token-ca-cert-hash sha256:da3ffd7e60b1ad55f2a95ea6375f15deb21d09299eb0b803b9f7fb8a4ad0a356

3.4 配置kubectl命令

为了能够使用kubectl，需要依赖/etc/kubernetes/admin.conf 文件，配置方法
在当前的登陆用户名下，新建一个.kube的目录，kubectl默认会读取这个文件夹下的配置，然后执行下面的复制命令

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

此时，就能执行kubectl的相关命令

[root@k8sMaster ~]# kubectl
kubectl controls the Kubernetes cluster manager.Find more information at:
https://kubernetes.io/docs/reference/kubectl/overview/Basic Commands (Beginner):create        Create a resource from a file or from stdin.expose        Take a replication controller, service, deployment or pod and
expose it as a new Kubernetes Servicerun           Run a particular image on the clusterset           Set specific features on objectsBasic Commands (Intermediate):explain       Documentation of resourcesget           Display one or many resourcesedit          Edit a resource on the serverdelete        Delete resources by filenames, stdin, resources and names, or by
resources and label selectorDeploy Commands:rollout       Manage the rollout of a resourcescale         Set a new size for a Deployment, ReplicaSet or Replication
Controllerautoscale     Auto-scale a Deployment, ReplicaSet, or ReplicationControllerCluster Management Commands:certificate   Modify certificate resources.cluster-info  Display cluster infotop           Display Resource (CPU/Memory/Storage) usage.cordon        Mark node as unschedulableuncordon      Mark node as schedulabledrain         Drain node in preparation for maintenancetaint         Update the taints on one or more nodesTroubleshooting and Debugging Commands:describe      Show details of a specific resource or group of resourceslogs          Print the logs for a container in a podattach        Attach to a running containerexec          Execute a command in a containerport-forward  Forward one or more local ports to a podproxy         Run a proxy to the Kubernetes API servercp            Copy files and directories to and from containers.auth          Inspect authorizationdebug         Create debugging sessions for troubleshooting workloads and
nodesAdvanced Commands:diff          Diff live version against would-be applied versionapply         Apply a configuration to a resource by filename or stdinpatch         Update field(s) of a resourcereplace       Replace a resource by filename or stdinwait          Experimental: Wait for a specific condition on one or many
resources.kustomize     Build a kustomization target from a directory or a remote url.Settings Commands:label         Update the labels on a resourceannotate      Update the annotations on a resourcecompletion    Output shell completion code for the specified shell (bash or
zsh)Other Commands:api-resources Print the supported API resources on the serverapi-versions  Print the supported API versions on the server, in the form of
"group/version"config        Modify kubeconfig filesplugin        Provides utilities for interacting with plugins.version       Print the client and server version informationUsage:kubectl [flags] [options]Use "kubectl <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all
commands).

3.5 配置Master节点的网络

wget https://docs.projectcalico.org/v3.8/manifests/calico.yaml
kubectl apply -f calico.yaml

3.6 安装Dashboard

3.6.1 安装Dashboard

安装Dashboard的命令行如下：

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yaml
kubectl apply -f recommended.yaml
kubectl get pods --all-namespaces | grep dashboard
kubernetes-dashboard   dashboard-metrics-scraper-7445d59dfd-pvr5q   1/1     Running   3          6d23h
kubernetes-dashboard   kubernetes-dashboard-5d6fdccd5-l5bk7         1/1     Running   3          6d23h

3.6.2 创建Dashboard用户

创建一个用于登录 Dashboard 的用户. 创建文件dashboard-adminuser.yaml内容如下:

apiVersion: v1
kind: ServiceAccount
metadata:name: admin-usernamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: admin-user
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: cluster-admin
subjects:
- kind: ServiceAccountname: admin-usernamespace: kube-system

并执行下面的命令

kubectl apply -f dashboard-adminuser.yaml

3.6.3 生成Dashboard证书

grep 'client-certificate-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.crt
grep 'client-key-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.key
openssl pkcs12 -export -clcerts -inkey kubecfg.key -in kubecfg.crt -out kubecfg.p12 -name "kubernetes-client"

3.6.4 登陆Dashboard

打开 Dashboard的时候，会首先进入下面的认证的页面。

一般我们会使用第二种方式，就是获取到Token，其命令如下

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

拷贝上面Token的值，注意不要包括前面的空格

至此，所有的Docker的列表如下：

4. 配置Worker节点并加入Master节点

Worker节点的加入比较简单，直接执行上面提到的 “kubeadm join ” 命令

kubeadm join 172.24.137.71:6443 --token 6s1u3g.rns2x03czqjux1x1 \--discovery-token-ca-cert-hash sha256:da3ffd7e60b1ad55f2a95ea6375f15deb21d09299eb0b803b9f7fb8a4ad0a356

5. 常见问题

5.1 Worker节点执行kubectl失败

k8s集群slave节点使用kubectl命令时The connection to the server localhost:8080 was refused - did you specify the right host or port?
mkdir -p /root/.kube

scp root@172.24.137.71:/root/.kube/* /root/.kube 或者 cp -i /etc/kubernetes/admin.kubeconfig /root/.kube/config

kubectl get nodes

5.2 Dashboard在公网环境下不能访问登陆页面

Dashboard在公网环境下不能打开下面的访问登陆页面，

解决方案：
在Master节点执行下面的命令
kubectl create clusterrolebinding test:anonymous --clusterrole=cluster-admin --user=system:anonymous

5.3 在Join Master节点的时候，一直在“pre flight”命令行下不动

Worker节点在加入Master节点时，一直在“pre flight”命令行下不动

kubeadm join 172.24.137.71:6443 --token 6s1u3g.rns2x03czqjux1x1 \--discovery-token-ca-cert-hash sha256:da3ffd7e60b1ad55f2a95ea6375f15deb21d09299eb0b803b9f7fb8a4ad0a356

解决办法：原因是Master生成的Token过期了
登陆到Master服务器上
3.1 原因是第一次生成的Token 过期了

 kubeadm token list   //token列表 ，第一次登陆上去，应该列表是空的kubeadm token create --ttl 0   //永不失效   kubeadm token create  //有效期一天

3.2 看出CA Token，一般CA Token是不会过期的

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
46f6cf1d84d0eadb4f6e7f05b908e5572025886d9f134db27f92b98e1c3dd3ed   //token

3.3 重新生成新的Worker加入Master节点的命令

kubeadm join 172.24.137.71:6443 --token abcdef.0123456789abcdef  \--discovery-token-ca-cert-hash sha256:da3ffd7e60b1ad55f2a95ea6375f15deb21d09299eb0b803b9f7fb8a4ad0a356

参考文档

kubernetes join 卡住 token过期
Kubernetes(一) 跟着官方文档从零搭建K8S