基于Kubernetes 1.22搭建基准测试系统(train-ticket)手册

本手册需要在搭建好kubernetes的基础上，再去运行基准测试系统。由于基准测试系统会占用大量的资源，所以要再真实的机器上搭建，或者是云主机上。

一、实验环境

二、环境检查

三、安装CRI-这里使用Docker

四、安装kubectl kubeadm kubelet

五、安装集群

六、部署基准测试系统

七、引用参考连接

一、实验环境

服务器系统	内核版本	节点IP	节点名称
Ubuntu 22.04	Linux 5.15.0-43-generic	192.168.105.132	master
Ubuntu 22.04	Linux 5.15.0-43-generic	192.168.105.133	worker-01
Ubuntu 22.04	Linux 5.15.0-43-generic	192.168.105.134	worker-02

二、环境检查

检查3台机器是否互相联通，并且均正常上网。

设置主机名

vim /etc/hostname # 命令

 # 192.168.105.132 主机修改为 master# 192.168.105.133 主机修改为 worker-01# 192.168.105.134 主机修改为 worker-01

主机名与IP进行映射

Note-1: 以下步骤需要在三台机器上执行，执行完成后重启服务器。

Note-2：修改完成后重启服务器。使用命令ping master 或 worker-01检查网络是否联通。

vim /etc/hosts # 命令

 127.0.0.1 localhost# 此处也需要修改为主机名127.0.1.1 master # 添加映射192.168.105.132 master192.168.105.133 worker-01192.168.105.134 worker-01# The following lines are desirable for IPv6 capable hosts::1     ip6-localhost ip6-loopbackfe00::0 ip6-localnetff00::0 ip6-mcastprefixff02::1 ip6-allnodesff02::2 ip6-allrouters

禁用swap

swap分区：交换分区，从磁盘里分一块空间来充当内存使用，性能比真正的物理内存要差。 docker容器在内存里运行，k8s不允许容器到swap分区运行，要关闭swap分区，所以关闭swap分区是k8s为了追求高性能。

(1)检查swap

 root@woker-01:~# free -htotal        used        free      shared  buff/cache   availableMem:           125Gi       1.3Gi       123Gi       5.0Mi       312Mi       123GiSwap:          8.0Gi          0B       8.0Gi

swap不为0,说明未禁用swap交换区

(2)临时关闭swap

sudo swapoff -a

再一次检查swap

 root@woker-01:~# free -htotal        used        free      shared  buff/cache   availableMem:           125Gi       1.3Gi       123Gi       5.0Mi       313Mi       123GiSwap:             0B          0B          0B

(3)永久关闭swap

 vim /etc/fstab/dev/disk/by-id/dm-uuid-LVM-r61BVsAoX3nEuGdY0vbFEnil06zm7gxIdciulVrVkn5zLvW7iRSXr5b14zeM3nnj / ext4 defaults 0 1# /boot was on /dev/sda2 during curtin installation/dev/disk/by-uuid/eeeb35a5-0d56-4a0a-960b-468fdc6820d9 /boot ext4 defaults 0 1# /boot/efi was on /dev/sda1 during curtin installation/dev/disk/by-uuid/2D61-3C9B /boot/efi vfat defaults 0 1# 注释下面一行的内容#/swap.img      none    swap    sw      0       0

关闭SElinux

需要安装相关工具包才能关闭

 sudo apt-get install -y policycoreutils

 root@worker-02:~# setenforce 0  #关闭selinuxsetenforce: SELinux is disabled

将文件系统设置为可读写

 sudo mount -n -o remount,rw /

三、安装CRI-这里使用Docker

Note-1: 以下步骤需要在三台机器上执行。

卸载旧版本，旧版本的 Docker 被称为 docker、docker.io 或 docker-engine。如果安装了这些，请卸载它们
```
sudo apt-get remove docker docker-engine docker.io containerd runc
```

更新 apt 包索引并安装包以允许 apt 通过 HTTPS 使用存储库

 sudo apt-get update

 sudo apt-get install \apt-transport-https \ca-certificates \curl \gnupg \lsb-release

添加Docker官方的GPG密钥

 curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

设置稳定存储库

 echo \"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

安装 Docker 引擎更新apt包索引

(1).更新apt包索
```
 sudo apt-get update
```
(2)安装最新版本的Docker Engine和containerd
```
 sudo apt-get install docker-ce docker-ce-cli containerd.io
```
Note-2:根据基准测试系统(train-ticket)的特殊，未能兼容最新版本。所以参考下图，安装以下版本

查询存储库中可用版本

 apt-cache madison docker-ce

使用第二列中的版本字符串安装特定版本，例如 5:20.10.18~3-0~ubuntu-jammy

 docker-ce | 5:20.10.18~3-0~ubuntu-jammy | https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages

(3) 安装指定版本得docker （5:20.10.18~3-0~ubuntu-jammy）

 sudo apt-get install docker-ce=5:20.10.18~3-0~ubuntu-jammy docker-ce-cli=5:20.10.18~3-0~ubuntu-jammy containerd.io=1.5.11-1

(4) 测试部署docker 是否成功

 docker run hello-world

Unable to find image 'hello-world:latest' locallylatest: Pulling from library/hello-world719385e32844: Pull complete Digest: sha256:fc6cf906cbfa013e80938cdf0bb199fbdbb86d6e3e013783e5a766f50f5dbce0Status: Downloaded newer image for hello-world:latestHello from Docker!This message shows that your installation appears to be working correctly.To generate this message, Docker took the following steps:1. The Docker client contacted the Docker daemon.2. The Docker daemon pulled the "hello-world" image from the Docker Hub.(amd64)3. The Docker daemon created a new container from that image which runs theexecutable that produces the output you are currently reading.4. The Docker daemon streamed that output to the Docker client, which sent itto your terminal.To try something more ambitious, you can run an Ubuntu container with:$ docker run -it ubuntu bashShare images, automate workflows, and more with a free Docker ID:https://hub.docker.com/For more examples and ideas, visit:https://docs.docker.com/get-started/

6.使普通用户也可以执行docker命令

 sudo gpasswd -a $USER docker #将登陆用户加入到docker用户组中newgrp docker #更新用户组

7.检查套接字

 ls /var/run/docker.sock

四、安装kubectl kubeadm kubelet

1.安装curl和apt-transport-https

sudo apt-get update && sudo apt-get install -y apt-transport-https curl

2.下载GPG

 sudo wget https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg

3.添加GPG

 sudo apt-key add apt-key.gpg

4.写入镜像源文件

注：没有此目录，直接在下面创建这个文件，并增加以下内容。

 sudo cat <<EOF >/etc/apt/sources.list.d/kubernetes.listdeb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial mainEOF

5.更新索引

 sudo apt-get update

6.安装kubectl kubeadm kubelet（安装指定版本1.22.4）

 sudo apt-get install kubeadm=1.22.4-00 kubelet=1.22.4-00 kubectl=1.22.4-00 -y

五、安装集群

1.检查需要哪些镜像

 root@master:~# kubeadm config images listI0511 10:00:28.341332    9281 version.go:255] remote version is much newer: v1.27.1; falling back to: stable-1.22k8s.gcr.io/kube-apiserver:v1.22.17k8s.gcr.io/kube-controller-manager:v1.22.17k8s.gcr.io/kube-scheduler:v1.22.17k8s.gcr.io/kube-proxy:v1.22.17k8s.gcr.io/pause:3.5k8s.gcr.io/etcd:3.5.0-0k8s.gcr.io/coredns/coredns:v1.8.4

2.拉取镜像文件到本地

 vim push.sh  # 执行命令# 将以下内容写入文件#!/bin/shsudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.22.4sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.22.4sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.22.4sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.22.4sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.0-0sudo docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4

 sudo chmod +x push.sh # 赋予执行权限./push.sh   # 拉取镜像

3.查看镜像文件

4.修改镜像名

vim tag.sh  # 执行命令# 将以下内容写入文件#!/bin/shsudo docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.22.17 k8s.gcr.io/kube-apiserver:v1.22.17sudo docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.22.17 k8s.gcr.io/kube-controller-manager:v1.22.17sudo docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.22.17 k8s.gcr.io/kube-scheduler:v1.22.17sudo docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.22.17 k8s.gcr.io/kube-proxy:v1.22.17sudo docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5 k8s.gcr.io/pause:3.5sudo docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.0-0 k8s.gcr.io/etcd:3.5.0-0sudo docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4 k8s.gcr.io/coredns/coredns:v1.8.4

 sudo chmod +x tag.sh # 赋予执行权限./tag.sh

5.修改docker默认的Cgroup Driver

(1).Kubernetes 推荐使用 systemd 来代替 cgroupfs。systemd是Kubernetes自带的cgroup管理器, 负责为每个进程分配cgroups。docker的cgroup driver默认是cgroupfs,这样就同时运行有两个cgroup控制管理器。当资源有压力的情况时,有可能出现不稳定的情况

 vim /etc/docker/daemon.json

 # 将以下内容写入文件{"exec-opts": ["native.cgroupdriver=systemd"]}

(2).重启docker

 sudo systemctl restart docker

(3).重启kubelet

 sudo systemctl restart kubelet

6.配置集群

(1).master执行--初始化集群

 kubeadm init --apiserver-advertise-address=192.168.105.132 --image-repository=registry.aliyuncs.com/google_containers --kubernetes-version=v1.22.17 --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all --v=6

初始化完成显示

(2).迁移配置

 mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config

(3)工作节点(worker)加入集群

 kubeadm join 172.16.2.146:6443 --token j4gpc3.l19pwajnb3xb60nu \--discovery-token-ca-cert-hash sha256:fc7649fda9a81f88e0089e8bf4df51cc119ee9d30038d4afc4bf058cba7cf362

查看节点状态

 root@master:~# kubectl get nodeNAME             STATUS     ROLES                  AGE    VERSIONgzdx-master      NotReady   control-plane,master   100s   v1.22.4gzdx-worker-01   NotReady   <none>                 21s    v1.22.4gzdx-worker-02   NotReady   <none>                 11s    v1.22.4

Note: 由于是低版本的kubernetes，所以节点状态为NotReady是正常的。原因是需要配置网络插件Calico.

7.配置集群网络插件

(1).下载官方Calico

 wget https://docs.projectcalico.org/v3.20/manifests/calico.yaml

(2).修改calico.yaml

 vim calico.yaml +3888# 大约在3888行，取消注释，并改为前面kubeadm init中pod-network-cidr字段定义的网段...- name: CALICO_IPV4POOL_CIDRvalue: "10.244.0.0/16"...

(3)修改完后应用清单

 kubectl apply -f calico.yaml

查看节点状态

 kubectl get no

查看Pod状态

 kubectl get pods -A

8.部署Nginx,检测集群是否正常

(1).在master 节点上创建一个deployment

 kubectl create deployment nginx --image=nginx

(2).创建一个service

 kubectl create service nodeport nginx --tcp 80:80

(3).查看Nginx服务

 oot@master:~# kubectl get svcNAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGEkubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        4m52snginx        NodePort    10.111.135.150   <none>        80:32608/TCP   2m31s

(4)访问Nginx服务

root@master:~# curl localhost:32608<!DOCTYPE html><html><head><title>Welcome to nginx!</title><style>html { color-scheme: light dark; }body { width: 35em; margin: 0 auto;font-family: Tahoma, Verdana, Arial, sans-serif; }</style></head><body><h1>Welcome to nginx!</h1><p>If you see this page, the nginx web server is successfully installed andworking. Further configuration is required.</p><p>For online documentation and support please refer to<a href="http://nginx.org/">nginx.org</a>.<br/>Commercial support is available at<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p></body></html>

至此集群部署成功。

六、部署基准测试系统

1.配置helm

wget https://get.helm.sh/helm-v3.10.3-linux-amd64.tar.gztar -zxvf helm-v3.10.3-linux-amd64.tar.gzmv linux-amd64/helm  /usr/local/bin/

执行helm list确认没有错误输出

 root@master:~# helm listNAME    NAMESPACE   REVISION    UPDATED STATUS  CHART   APP VERSION

2.安装OpenEBS

(1)先决条件

1). 确认 master 节点是否有 Taint，如下看到 master 节点有 Taint

 root@master:~# kubectl describe node gzdx-master | grep TaintTaints:             node-role.kubernetes.io/master:NoSchedule

2).暂时去掉 master 节点的 Taint

 kubectl taint nodes gzdx-master node-role.kubernetes.io/master:NoSchedule-

(2).使用 Helm 安装openebs

 root@master:~# helm repo add openebs https://openebs.github.io/charts"openebs" has been added to your repositoriesroot@master:~# helm repo updateHang tight while we grab the latest from your chart repositories......Successfully got an update from the "openebs" chart repositoryUpdate Complete. ⎈Happy Helming!⎈

 root@master:~# helm repo listNAME    URL                             openebs https://openebs.github.io/charts

root@master:~# helm install openebs --namespace openebs openebs/openebs --create-namespace# 以下是成功后输出的内容NAME: openebsLAST DEPLOYED: Thu May 11 11:13:47 2023NAMESPACE: openebsSTATUS: deployedREVISION: 1TEST SUITE: NoneNOTES:Successfully installed OpenEBS.Check the status by running: kubectl get pods -n openebsThe default values will install NDM and enable OpenEBS hostpath and devicestorage engines along with their default StorageClasses. Use `kubectl get sc`to see the list of installed OpenEBS StorageClasses.**Note**: If you are upgrading from the older helm chart that was using cStorand Jiva (non-csi) volumes, you will have to run the following command to includethe older provisioners:helm upgrade openebs openebs/openebs \--namespace openebs \--set legacy.enabled=true \--reuse-valuesFor other engines, you will need to perform a few more additional steps toenable the engine, configure the engines (e.g. creating pools) and create StorageClasses. For example, cStor can be enabled using commands like:helm upgrade openebs openebs/openebs \--namespace openebs \--set cstor.enabled=true \--reuse-valuesFor more information, - view the online documentation at https://openebs.io/docs or- connect with an active community on Kubernetes slack #openebs channel

查看创建的 StorageClass

将设置为默认的 StorageClass openebs-hostpath

 kubectl patch storageclass openebs-hostpath -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

查看openebs状态

3.拉取系统

 git clone --depth=1 https://github.com/FudanSELab/train-ticket.git cd train-ticket/

4.部署系统

 make deploy

七、引用参考连接

1.FudanSELab/train-ticket: Train Ticket - A Benchmark Microservice System (github.com)

2.kubeadm快速搭建K8s集群基于v1.22.2版本 - 努力吧阿团 - 博客园 (cnblogs.com)

3.安装OpenEBS (timd.cn)

4.ubuntu 20.04 基于kubeadm部署kubernetes 1.22.4集群及部署集群管理工具 - 西瓜君~ - 博客园 (cnblogs.com)

5.使用kubeadm部署Kubernetes 1.26 - 架构小白|青蛙小白|关注程序开发、互联网技术、云原生 (frognew.com)

6.(18条消息) Kubernetes系列之三：部署你的第一个应用程序到k8s集群_wucong60的博客-CSDN博客