前言：k3s高可用还需要做个前段代理。后续更新上。

环境

操作系统内核

[root@rancher01 ~]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
[root@rancher01 ~]# uname -a
Linux rancher01 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

ip主机名

10.0.0.10 rancher01
10.0.0.10 rancher01
10.0.0.10 rancher01

docker未安装，防火墙及selinux关闭。

前期准备

安装mysql

rancher01节点安装mysql服务端及客户端，为rancher提供数据库服务。

[root@rancher01 ~]# yum -y install mysql mariadb-server

mysql设置密码

mysqladmin -u root password 123456

mysql修改密码（不操作）

mysqladmin -u root -p password abcdef

注意，命令回车后会问你旧密码，输入旧密码123456之后命令完成，密码修改成功。
创建数据库

MariaDB [(none)]> create database k3s;
Query OK, 1 row affected (0.00 sec)

mysql授权

MariaDB [(none)]> grant all privileges on *.* to root@'rancher01' identified by '123456';
Query OK, 0 rows affected (0.00 sec)MariaDB [(none)]> grant all privileges on *.* to root@'10.0.0.11' identified by "123456";
Query OK, 0 rows affected (0.00 sec)MariaDB [(none)]> grant all privileges on *.* to root@'10.0.0.11' identified by "123456";
Query OK, 0 rows affected (0.00 sec)MariaDB [(none)]> select host,user from mysql.user;
+-----------+------+
| host      | user |
+-----------+------+
| 10.0.0.10 | root |
| 10.0.0.11 | root |
| 10.0.0.12 | root |
| 127.0.0.1 | root |
| ::1       | root |
| localhost |      |
| localhost | root |
| rancher01 |      |
| rancher01 | root |
+-----------+------+
9 rows in set (0.00 sec)

rancher02节点安装mysql客户端，仅仅为了验证mysql登录权限，rancher连接mysql不需要安装mysql客户端。

[root@rancher02 ~]# yum -y install mysql

验证mysql登录权限

[root@rancher01 ~]# mysql -uroot -p123456 -h10.0.0.10
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 574
Server version: 5.5.68-MariaDB MariaDB ServerCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [(none)]> [root@rancher02 ~]# mysql -uroot -p123456 -h 10.0.0.10
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 2730
Server version: 5.5.68-MariaDB MariaDB ServerCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [(none)]> quit
Bye

安装k3s集群

rancher资源

http://mirror.cnrancher.com/

配置镜像仓库

http://docs.rancher.cn/docs/k3s/installation/private-registry/_index/

安装选项查看

https://docs.rancher.cn/docs/k3s/installation/install-options/_index/#%E4%BD%BF%E7%94%A8%E8%84%9A%E6%9C%AC%E5%AE%89%E8%A3%85%E7%9A%84%E9%80%89%E9%A1%B9

##需要的镜像文件

mkdir -p /var/lib/rancher/k3s/agent/images/

下载镜像包k3s-airgap-images-amd64.tar
下载地址

http://mirror.cnrancher.com/

[root@rancher01 ~]# cp k3s-airgap-images-amd64.tar /var/lib/rancher/k3s/agent/images/
[root@rancher01 ~]# scp k3s-airgap-images-amd64.tar 10.0.0.11://var/lib/rancher/k3s/agent/images/

[root@rancher01 ~]# cp k3s /usr/local/bin/
[root@rancher01 ~]# chmod +x /usr/local/bin/k3s
[root@rancher01 ~]# scp -rp k3s 10.0.0.11:/usr/local/bin/
[root@rancher02 ~]# chmod +x /usr/local/bin/k3s

部署

在线安装

rancher01节点执行

curl -sfL https://get.k3s.io | sh -s - server   --datastore-endpoint="mysql://root:123456@tcp(10.0.0.10:3306)/k3s"

使用国内地址
注意：pod镜像源地址并未改变。

curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -s - server --datastore-endpoint="mysql://root:123456@tcp(10.0.0.10:3306)/k3s"

rancher02节点执行
警告：必须先获取node_token，否则启动k3s会报错。
获取token

[root@rancher01 ~]# cat /var/lib/rancher/k3s/server/node-token
K10a5c63be28debef03924537b75d02b46c5859f21a873ff8b76981aae592a72802::server:49a7301f52785cdb2c48da86c9b6ed41

通过K3S_TOKEN参数指定token

[root@rancher02 ~]# curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_VERSION=v1.19.3 INSTALL_K3S_SKIP_DOWNLOAD=true K3S_TOKEN=$token INSTALL_K3S_MIRROR=cn sh -s - server --datastore-endpoint="mysql://root:123456@tcp(10.0.0.10:3306)/k3s"

离线安装

所有节点执行如下命令

INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC='server --datastore-endpoint=mysql://root:123456@tcp(10.0.0.10:3306)/k3s' ./k3s-install.sh

部署过程中输出

离线安装的输出

[INFO]  Skipping k3s download and verify
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
[INFO]  systemd: Starting k3s

在线安装的输出

[INFO]  Finding release for channel stable
[INFO]  Using v1.21.5+k3s2 as release
[INFO]  Downloading hash http://rancher-mirror.cnrancher.com/k3s/v1.21.5-k3s2/sha256sum-amd64.txt
[INFO]  Downloading binary http://rancher-mirror.cnrancher.com/k3s/v1.21.5-k3s2/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile* base: mirrors.bupt.edu.cn* extras: mirrors.bupt.edu.cn* updates: mirrors.bupt.edu.cn
Resolving Dependencies
--> Running transaction check
省略yum安装软件的输出
Complete!
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
[INFO]  systemd: Starting k3s

验证k3s

[root@rancher01 ~]# kubectl get nodes
NAME        STATUS   ROLES                  AGE   VERSION
rancher02   Ready    control-plane,master   87s   v1.21.5+k3s2
rancher01   Ready    control-plane,master   18m   v1.21.5+k3s2[root@rancher01 ~]# kubectl get pods -A -o wide
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE     IP          NODE        NOMINATED NODE   READINESS GATES
kube-system   metrics-server-86cbb8457f-2dsr7           1/1     Running     0          22m     10.42.0.4   rancher01   <none>           <none>
kube-system   coredns-7448499f4d-j5kk8                  1/1     Running     0          22m     10.42.0.3   rancher01   <none>           <none>
kube-system   local-path-provisioner-5ff76fc89d-2xgrk   1/1     Running     1          22m     10.42.0.2   rancher01   <none>           <none>
kube-system   helm-install-traefik-crd-bmvxg            0/1     Completed   0          22m     10.42.0.6   rancher01   <none>           <none>
kube-system   helm-install-traefik-r9w49                0/1     Completed   1          22m     10.42.0.5   rancher01   <none>           <none>
kube-system   svclb-traefik-qdpgk                       2/2     Running     0          4m47s   10.42.0.7   rancher01   <none>           <none>
kube-system   traefik-97b44b794-vsdnt                   1/1     Running     0          4m47s   10.42.1.2   rancher02   <none>           <none>
kube-system   svclb-traefik-xbcvk                       2/2     Running     0          4m47s   10.42.1.3   rancher02   <none>           <none>

k3s安装排错

数据库权限错误

执行结果

Complete!
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink from /etc/systemd/system/multi-user.target.wants/k3s.service to /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
Job for k3s.service failed because the control process exited with error code. See "systemctl status k3s.service" and "journalctl -xe" for details.

查看问题

[root@rancher01 ~]# journalctl -xe
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit k3s.service has begun starting up.
Nov 11 14:41:56 rancher01 sh[47129]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Nov 11 14:41:56 rancher01 sh[47129]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Nov 11 14:41:56 rancher01 k3s[47137]: time="2021-11-11T14:41:56.783415378+08:00" level=info msg="Starting k3s v1.21.5+k3s2 (724ef700)"
Nov 11 14:41:56 rancher01 systemd[1]: k3s.service: main process exited, code=exited, status=1/FAILURE
Nov 11 14:41:56 rancher01 k3s[47137]: time="2021-11-11T14:41:56.785100666+08:00" level=fatal msg="starting kubernetes: preparing server: creating storage endpoint: building kine: Error 1130
Nov 11 14:41:56 rancher01 systemd[1]: Failed to start Lightweight Kubernetes.
-- Subject: Unit k3s.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit k3s.service has failed.
--
-- The result is failed.
Nov 11 14:41:56 rancher01 systemd[1]: Unit k3s.service entered failed state.
Nov 11 14:41:56 rancher01 systemd[1]: k3s.service failed.
Nov 11 14:42:01 rancher01 systemd[1]: k3s.service holdoff time over, scheduling restart.
Nov 11 14:42:01 rancher01 systemd[1]: Starting Lightweight Kubernetes...
-- Subject: Unit k3s.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit k3s.service has begun starting up.
Nov 11 14:42:01 rancher01 sh[47161]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Nov 11 14:42:01 rancher01 sh[47161]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Nov 11 14:42:02 rancher01 k3s[47170]: time="2021-11-11T14:42:02.333994988+08:00" level=info msg="Starting k3s v1.21.5+k3s2 (724ef700)"
Nov 11 14:42:02 rancher01 k3s[47170]: time="2021-11-11T14:42:02.335762321+08:00" level=fatal msg="starting kubernetes: preparing server: creating storage endpoint: building kine: Error 1130
Nov 11 14:42:02 rancher01 systemd[1]: k3s.service: main process exited, code=exited, status=1/FAILURE
Nov 11 14:42:02 rancher01 systemd[1]: Failed to start Lightweight Kubernetes.
-- Subject: Unit k3s.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit k3s.service has failed.
--
-- The result is failed.
Nov 11 14:42:02 rancher01 systemd[1]: Unit k3s.service entered failed state.
Nov 11 14:42:02 rancher01 systemd[1]: k3s.service failed.
lines 2163-2201/2201 (END)

解决
没具体分析
查看详细问题命令，这里我没查看，初步判断为是数据库连接不上的问题。

journalctl -f -u k3s

[root@rancher01 ~]# mysql -uroot -h rancher01 -p123456
ERROR 1130 (HY000): Host 'rancher01' is not allowed to connect to this MariaDB server

token不一致错误

Complete!
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
[INFO]  systemd: Starting k3s
Job for k3s.service failed because the control process exited with error code. See "systemctl status k3s.service" and "journalctl -xe" for details.

[root@rancher02 ~]# journalctl -f -u k3s
-- Logs begin at Thu 2021-11-11 16:51:12 CST. --
Nov 11 17:18:12 rancher02 k3s[3732]: time="2021-11-11T17:18:12.469855660+08:00" level=info msg="Starting k3s v1.21.5+k3s2 (724ef700)"
Nov 11 17:18:12 rancher02 k3s[3732]: time="2021-11-11T17:18:12.490484887+08:00" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Nov 11 17:18:12 rancher02 k3s[3732]: time="2021-11-11T17:18:12.490558719+08:00" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Nov 11 17:18:12 rancher02 k3s[3732]: time="2021-11-11T17:18:12.502097066+08:00" level=info msg="Database tables and indexes are up to date"
Nov 11 17:18:12 rancher02 k3s[3732]: time="2021-11-11T17:18:12.518683948+08:00" level=info msg="Kine listening on unix://kine.sock"
Nov 11 17:18:12 rancher02 k3s[3732]: time="2021-11-11T17:18:12.562636903+08:00" level=fatal msg="starting kubernetes: preparing server: bootstrap data already found and encrypted with different token"
Nov 11 17:18:12 rancher02 systemd[1]: k3s.service: main process exited, code=exited, status=1/FAILURE
Nov 11 17:18:12 rancher02 systemd[1]: Failed to start Lightweight Kubernetes.
Nov 11 17:18:12 rancher02 systemd[1]: Unit k3s.service entered failed state.
Nov 11 17:18:12 rancher02 systemd[1]: k3s.service failed.

解决
去rancher01节点查看token

[root@rancher01 ~]# cat /var/lib/rancher/k3s/server/node-token
K10a5c63be28debef03924537b75d02b46c5859f21a873ff8b76981aae592a72802::server:49a7301f52785cdb2c48da86c9b6ed41

curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | K3S_TOKEN=K10a5c63be28debef03924537b75d02b46c5859f21a873ff8b76981aae592a72802::server:49a7301f52785cdb2c48da86c9b6ed41 INSTALL_K3S_MIRROR=cn sh -s - server --datastore-endpoint="mysql://root:123456@tcp(10.0.0.10:3306)/k3s"

安装rancher

安装helm

wget https://get.helm.sh/helm-v3.2.4-linux-amd64.tar.gz
tar -xf helm-v3.2.4-linux-amd64.tar.gz
mv linux-amd64/helm /usr/bin/

配置helm连接k3s

echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> /etc/profile && source /etc/profile

添加rancher镜像源

helm repo add rancher-stable http://rancher-mirror.oss-cn-beijing.aliyuncs.com/server-charts/stable
helm repo update

kubectl create namespace cattle-system

生成证书

官网地址

http://docs.rancher.cn/docs/rancher2/admin-settings/replace-ip-domain/_index/#%E6%AD%A5%E9%AA%A4-2%EF%BC%9A%E5%87%86%E5%A4%87%E8%AF%81%E4%B9%A6

创建证书目录

mkdir /certs
cd /certs

vim create_self-signed-cert.sh

#!/bin/bash -ehelp ()
{echo  ' ================================================================ 'echo  ' --ssl-domain: 生成ssl证书需要的主域名，如不指定则默认为www.rancher.local，如果是ip访问服务，则可忽略；'echo  ' --ssl-trusted-ip: 一般ssl证书只信任域名的访问请求，有时候需要使用ip去访问Server，那么需要给ssl证书添加扩展IP，多个IP用逗号隔开；'echo  ' --ssl-trusted-domain: 如果想多个域名访问，则添加扩展域名（SSL_TRUSTED_DOMAIN）,多个扩展域名用逗号隔开；'echo  ' --ssl-size: ssl加密位数，默认2048；'echo  ' --ssl-cn: 国家代码(2个字母的代号),默认CN;'echo  ' 使用示例:'echo  ' ./create_self-signed-cert.sh --ssl-domain=www.test.com --ssl-trusted-domain=www.test2.com \ 'echo  ' --ssl-trusted-ip=1.1.1.1,2.2.2.2,3.3.3.3 --ssl-size=2048 --ssl-date=3650'echo  ' ================================================================'
}case "$1" in-h|--help) help; exit;;
esacif [[ $1 == '' ]];thenhelp;exit;
fiCMDOPTS="$*"
for OPTS in $CMDOPTS;
dokey=$(echo ${OPTS} | awk -F"=" '{print $1}' )value=$(echo ${OPTS} | awk -F"=" '{print $2}' )case "$key" in--ssl-domain) SSL_DOMAIN=$value ;;--ssl-trusted-ip) SSL_TRUSTED_IP=$value ;;--ssl-trusted-domain) SSL_TRUSTED_DOMAIN=$value ;;--ssl-size) SSL_SIZE=$value ;;--ssl-date) SSL_DATE=$value ;;--ca-date) CA_DATE=$value ;;--ssl-cn) CN=$value ;;esac
done# CA相关配置
CA_DATE=${CA_DATE:-3650}
CA_KEY=${CA_KEY:-cakey.pem}
CA_CERT=${CA_CERT:-cacerts.pem}
CA_DOMAIN=cattle-ca# ssl相关配置
SSL_CONFIG=${SSL_CONFIG:-$PWD/openssl.cnf}
SSL_DOMAIN=${SSL_DOMAIN:-'www.rancher.local'}
SSL_DATE=${SSL_DATE:-3650}
SSL_SIZE=${SSL_SIZE:-2048}## 国家代码(2个字母的代号),默认CN;
CN=${CN:-CN}SSL_KEY=$SSL_DOMAIN.key
SSL_CSR=$SSL_DOMAIN.csr
SSL_CERT=$SSL_DOMAIN.crtecho -e "\033[32m ---------------------------- \033[0m"
echo -e "\033[32m       | 生成 SSL Cert |       \033[0m"
echo -e "\033[32m ---------------------------- \033[0m"if [[ -e ./${CA_KEY} ]]; thenecho -e "\033[32m ====> 1. 发现已存在CA私钥，备份"${CA_KEY}"为"${CA_KEY}"-bak，然后重新创建 \033[0m"mv ${CA_KEY} "${CA_KEY}"-bakopenssl genrsa -out ${CA_KEY} ${SSL_SIZE}
elseecho -e "\033[32m ====> 1. 生成新的CA私钥 ${CA_KEY} \033[0m"openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
fiif [[ -e ./${CA_CERT} ]]; thenecho -e "\033[32m ====> 2. 发现已存在CA证书，先备份"${CA_CERT}"为"${CA_CERT}"-bak，然后重新创建 \033[0m"mv ${CA_CERT} "${CA_CERT}"-bakopenssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
elseecho -e "\033[32m ====> 2. 生成新的CA证书 ${CA_CERT} \033[0m"openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
fiecho -e "\033[32m ====> 3. 生成Openssl配置文件 ${SSL_CONFIG} \033[0m"
cat > ${SSL_CONFIG} <<EOM
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOMif [[ -n ${SSL_TRUSTED_IP} || -n ${SSL_TRUSTED_DOMAIN} ]]; thencat >> ${SSL_CONFIG} <<EOM
subjectAltName = @alt_names
[alt_names]
EOMIFS=","dns=(${SSL_TRUSTED_DOMAIN})dns+=(${SSL_DOMAIN})for i in "${!dns[@]}"; doecho DNS.$((i+1)) = ${dns[$i]} >> ${SSL_CONFIG}doneif [[ -n ${SSL_TRUSTED_IP} ]]; thenip=(${SSL_TRUSTED_IP})for i in "${!ip[@]}"; doecho IP.$((i+1)) = ${ip[$i]} >> ${SSL_CONFIG}donefi
fiecho -e "\033[32m ====> 4. 生成服务SSL KEY ${SSL_KEY} \033[0m"
openssl genrsa -out ${SSL_KEY} ${SSL_SIZE}echo -e "\033[32m ====> 5. 生成服务SSL CSR ${SSL_CSR} \033[0m"
openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} -subj "/C=${CN}/CN=${SSL_DOMAIN}" -config ${SSL_CONFIG}echo -e "\033[32m ====> 6. 生成服务SSL CERT ${SSL_CERT} \033[0m"
openssl x509 -sha256 -req -in ${SSL_CSR} -CA ${CA_CERT} \-CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \-days ${SSL_DATE} -extensions v3_req \-extfile ${SSL_CONFIG}echo -e "\033[32m ====> 7. 证书制作完成 \033[0m"
echo
echo -e "\033[32m ====> 8. 以YAML格式输出结果 \033[0m"
echo "----------------------------------------------------------"
echo "ca_key: |"
cat $CA_KEY | sed 's/^/  /'
echo
echo "ca_cert: |"
cat $CA_CERT | sed 's/^/  /'
echo
echo "ssl_key: |"
cat $SSL_KEY | sed 's/^/  /'
echo
echo "ssl_csr: |"
cat $SSL_CSR | sed 's/^/  /'
echo
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/  /'
echoecho -e "\033[32m ====> 9. 附加CA证书到Cert文件 \033[0m"
cat ${CA_CERT} >> ${SSL_CERT}
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/  /'
echoecho -e "\033[32m ====> 10. 重命名服务证书 \033[0m"
echo "cp ${SSL_DOMAIN}.key tls.key"
cp ${SSL_DOMAIN}.key tls.key
echo "cp ${SSL_DOMAIN}.crt tls.crt"
cp ${SSL_DOMAIN}.crt tls.crt

chmod +x create_self-signed-cert.sh

sh ./create_self-signed-cert.sh --ssl-domain=zhou.rancher.com --ssl-trusted-ip=192.168.15.251,192.168.15.252 --ssl-size=2048 --ssl-date=3650

上传证书到k8s集群

把生成的pem证书上传到集群中去
ingress要用

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl --kubeconfig=$kubeconfig create namespace cattle-system
kubectl --kubeconfig=$kubeconfig \-n cattle-system create \secret tls tls-rancher-ingress \--cert=./tls.crt \--key=./tls.key

rancher要用

kubectl -n cattle-system create secret generic tls-ca   --from-file=cacerts.pem=./cacerts.pem

helm安装rancher

安装选项查看

https://docs.rancher.cn/docs/rancher2.5/installation/install-rancher-on-k8s/chart-options/_index/

方式 C：使用您已有的证书#
在此选项中，将使用您自己的证书来创建 Kubernetes 密文，以供 Rancher 使用。

当您运行此命令时，hostname选项必须与服务器证书中的Common Name或Subject Alternative Names条目匹配，否则 Ingress 控制器将无法正确配置。

尽管技术上仅需要Subject Alternative Names中有一个条目，但是拥有一个匹配的 Common Name 可以最大程度的提高与旧版浏览器/应用程序的兼容性。

如果您想检查证书是否正确，请查看如何在服务器证书中检查 Common Name 和 Subject Alternative Names。

如上所述，为你的证书设置适当的hostname。
将replicas设置为 Rancher 部署所使用的复制数量。默认为 3；如果你的集群中少于 3 个节点，你应填写实际节点数量。
设置ingress.tls.source为secret。
要安装一个特定的 Rancher 版本，使用–version 标志，例如：–version 2.3.6。
如果你安装的是 alpha 版本，Helm 要求在命令中加入–devel选项。

[root@rancher01 certs]# pwd
/certs
[root@rancher01 certs]# openssl x509 -noout -subject -in cacerts.pem
subject= /C=CN/CN=cattle-ca

helm install rancher rancher-stable/rancher   --namespace cattle-system   --set hostname=zhou.rancher.com     --set ingress.tls.source=secret    --set rancherImage=registry.cn-hangzhou.aliyuncs.com/rancher/rancher --set replicas=2 --set privateCA=true

注意：这里的–set hostname值要和执行创建证书脚本时指定的–ssl-domain值一致。

添加DNS解析

说明：在要访问rancher的客户端也添加解析，比如windows的hosts文件里。这里在部署rancher的服务器添加解析。

[root@rancher01 certs]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.10 rancher01
10.0.0.11 rancher02
192.168.103.88 zhou.rancher.com

查看rancher资源

[root@rancher01 certs]# kubectl get ingress -A
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAMESPACE       NAME      CLASS    HOSTS              ADDRESS          PORTS     AGE
cattle-system   rancher   <none>   zhou.rancher.com   192.168.103.88   80, 443   3h23m[root@rancher01 ~]# kubectl get pods -A
NAMESPACE                   NAME                                     READY   STATUS      RESTARTS   AGE
kube-system                 helm-install-traefik-h6gml               0/1     Completed   0          7h7m
cattle-system               rancher-5f4496bfbb-kk6s9                 1/1     Running     0          4h25m
cattle-fleet-local-system   fleet-agent-59b74595c-2rdzf              1/1     Running     0          3h46m
kube-system                 metrics-server-7b4f8b595-wrxtg           1/1     Running     2          7h7m
cattle-fleet-system         fleet-controller-66cc4c6b5b-qn57v        1/1     Running     0          4h25m
cattle-system               rancher-webhook-6979fbd4bf-5rdmc         1/1     Running     0          4h24m
kube-system                 local-path-provisioner-7ff9579c6-x2sgn   1/1     Running     9          7h7m
cattle-fleet-system         gitjob-5778966b7c-xt5ww                  1/1     Running     0          4h25m
kube-system                 traefik-5dd496474-jz58j                  1/1     Running     1          7h7m
cattle-system               rancher-5f4496bfbb-g7rkz                 1/1     Running     0          4h26m
kube-system                 coredns-66c464876b-kvgz4                 1/1     Running     1          7h7m
kube-system                 svclb-traefik-fkv8m                      2/2     Running     2          7h7m
[root@rancher01 ~]# kubectl get svc -A
NAMESPACE             NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
default               kubernetes           ClusterIP      10.43.0.1       <none>           443/TCP                      7h8m
kube-system           kube-dns             ClusterIP      10.43.0.10      <none>           53/UDP,53/TCP,9153/TCP       7h8m
kube-system           metrics-server       ClusterIP      10.43.174.191   <none>           443/TCP                      7h8m
kube-system           traefik-prometheus   ClusterIP      10.43.15.56     <none>           9100/TCP                     7h7m
cattle-system         rancher              ClusterIP      10.43.131.154   <none>           80/TCP,443/TCP               6h14m
cattle-fleet-system   gitjob               ClusterIP      10.43.78.229    <none>           80/TCP                       4h25m
cattle-system         webhook-service      ClusterIP      10.43.119.56    <none>           443/TCP                      4h24m
cattle-system         rancher-webhook      ClusterIP      10.43.235.111   <none>           443/TCP                      4h24m
kube-system           traefik              LoadBalancer   10.43.43.110    192.168.103.88   80:30457/TCP,443:30758/TCP   7h7m
[root@rancher01 ~]# kubectl get deploy -A
NAMESPACE                   NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
cattle-fleet-local-system   fleet-agent              1/1     1            1           4h14m
kube-system                 metrics-server           1/1     1            1           7h8m
cattle-fleet-system         fleet-controller         1/1     1            1           4h25m
cattle-system               rancher-webhook          1/1     1            1           4h24m
kube-system                 local-path-provisioner   1/1     1            1           7h8m
cattle-fleet-system         gitjob                   1/1     1            1           4h25m
kube-system                 traefik                  1/1     1            1           7h7m
cattle-system               rancher                  2/2     2            2           6h14m
kube-system                 coredns                  1/1     1            1           7h8m

验证rancher

[root@rancher01 ~]# kubectl -n cattle-system rollout status deploy/rancher
deployment "rancher" successfully rolled out
[root@rancher01 ~]# kubectl -n cattle-system get deploy rancher
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
rancher   2/2     2            2           6h13m

curl rancher域名（即证书中设置的域名）
注意：curl ip是不通的，只有curl域名才通。

[root@rancher01 certs]# curl zhou.rancher.com
<a href="https://zhou.rancher.com/">Found</a>.

浏览器访问

kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}'

rancher安装排错

helm连接k3s集群错误

[root@rancher01 certs]# helm install rancher rancher-stable/rancher   --namespace cattle-system   --set hostname=zhou.rancher.com   --set ingress.tls.source=secret   --set privateCA=true
Error: Kubernetes cluster unreachable[root@rancher01 certs]# helm  list
Error: Kubernetes cluster unreachable

解决
临时解决:

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

永久解决:

echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> /etc/profile && source /etc/profile

mysql连接数错误

[root@rancher01 ~]# kubectl get pods -A
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?

[root@rancher01 ~]# journalctl -f -u k3s
-- Logs begin at Wed 2021-11-17 13:26:25 CST. --
Nov 17 14:23:25 rancher01 systemd[1]: k3s.service failed.
Nov 17 14:23:30 rancher01 systemd[1]: k3s.service holdoff time over, scheduling restart.
Nov 17 14:23:30 rancher01 systemd[1]: Starting Lightweight Kubernetes...
Nov 17 14:23:30 rancher01 k3s[28280]: time="2021-11-17T14:23:30.705632784+08:00" level=info msg="Starting k3s v1.19.3+k3s2 (f8a4547b)"
Nov 17 14:23:30 rancher01 k3s[28280]: time="2021-11-17T14:23:30.707945642+08:00" level=info msg="Cluster bootstrap already complete"
Nov 17 14:23:30 rancher01 k3s[28280]: time="2021-11-17T14:23:30.722104947+08:00" level=fatal msg="starting kubernetes: preparing server: creating storage endpoint: building kine: Error 1129: Host '10.0.0.10' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'"
Nov 17 14:23:30 rancher01 systemd[1]: k3s.service: main process exited, code=exited, status=1/FAILURE
Nov 17 14:23:30 rancher01 systemd[1]: Failed to start Lightweight Kubernetes.
Nov 17 14:23:30 rancher01 systemd[1]: Unit k3s.service entered failed state.
Nov 17 14:23:30 rancher01 systemd[1]: k3s.service failed.
Nov 17 14:23:35 rancher01 systemd[1]: k3s.service holdoff time over, scheduling restart.
Nov 17 14:23:35 rancher01 systemd[1]: Starting Lightweight Kubernetes...
Nov 17 14:23:36 rancher01 k3s[28295]: time="2021-11-17T14:23:36.181898604+08:00" level=info msg="Starting k3s v1.19.3+k3s2 (f8a4547b)"
Nov 17 14:23:36 rancher01 k3s[28295]: time="2021-11-17T14:23:36.183024018+08:00" level=info msg="Cluster bootstrap already complete"
Nov 17 14:23:36 rancher01 k3s[28295]: time="2021-11-17T14:23:36.198187738+08:00" level=fatal msg="starting kubernetes: preparing server: creating storage endpoint: building kine: Error 1129: Host '10.0.0.10' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'"
Nov 17 14:23:36 rancher01 systemd[1]: k3s.service: main process exited, code=exited, status=1/FAILURE
Nov 17 14:23:36 rancher01 systemd[1]: Failed to start Lightweight Kubernetes.
Nov 17 14:23:36 rancher01 systemd[1]: Unit k3s.service entered failed state.
Nov 17 14:23:36 rancher01 systemd[1]: k3s.service failed.

解决

vim /usr/lib/systemd/system/mariadb.service
[Service]
LimitNOFILE=16384
LimitNPROC=16384vim /etc/my.cnf
[mysqld]
max_connections=16384

systemctl daemon-reload
systemctl restart mariadb

racnher找不到证书错误

kubectl describe pod `kubectl get pods -A |grep rancher|head -1|awk -F '[ ]+' '{print $2}'` -n cattle-system
部分输出省略
Events:Type     Reason       Age                   From               Message----     ------       ----                  ----               -------Normal   Scheduled    4m44s                 default-scheduler  Successfully assigned cattle-system/rancher-5f4496bfbb-z9lqg to rancher01Warning  FailedMount  34s (x10 over 4m44s)  kubelet            MountVolume.SetUp failed for volume "tls-ca-volume" : secret "tls-ca" not foundWarning  FailedMount  26s (x2 over 2m41s)   kubelet            Unable to attach or mount volumes: unmounted volumes=[tls-ca-volume], unattached volumes=[tls-ca-volume rancher-token-5kwbn]: timed out waiting for the condition

解决

cd /certs/
说明：该证书生成方式见上文 生成证书
kubectl -n cattle-system create secret generic tls-ca   --from-file=cacerts.pem=./cacerts.pem

访问域名404错误

[root@rancher01 certs]# curl zhou.rancher.com
404 page not found

排查问题

spec:                                                                          rules:                                                                       - host: zhou.rancher.com                                                     http:                                                                      paths:                               - backend:          serviceName: rancherservicePort: 80     pathType: ImplementationSpecifictls:                                  - hosts:                              - zhou.rancher.com                  ~~secretName: tls-rancher-ingress~~     ~~ **

解决

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl --kubeconfig=$kubeconfig create namespace cattle-system
kubectl --kubeconfig=$kubeconfig \-n cattle-system create \secret tls tls-rancher-ingress \--cert=./tls.crt \--key=./tls.key

K3s部署rancher相关推荐

K3s集群部署+rancher部署并导入K3s集群
centos系统自动部署docker: curl -fsSL get.docker.com | sh 运用国内的资源安装 K3s: K3s server安装(脚本安装): curl –sfL \ ht ...
使用k3s部署轻量Kubernetes集群快速教程
k3s是轻量级的Kubernetes.安装简单,占用资源少,只需要512M内存就可以运行起来,所有的二进制程序都不到 100MB. 自 2019 年 3 月发布以来,备受全球开发者们关注.至今,Git ...
docker、K8S、k3s、rancher
第一:docker 基础使用 docker 竞品最新调查:OpenStack.Docker.KVM被评为最火的云开源项目. Docker替代品,Containerd Docker与KVM之间的区别( ...
RKE部署Rancher v2.5.8 HA高可用集群以及常见错误解决
此博客,是根据Rancher官网文档,使用RKE测试部署最新发布版 Rancher v2.5.8 高可用集群的总结文档.Rancher文档 | K8S文档 | Rancher | Rancher文档 ...
使用RKE部署Rancher v2.5.8 HA高可用集群
文章目录一了解 Rancher 1 关于Helm 2 关于RKE 3 关于K3S 4 Rancher 名词解释 4.1 仪表盘 4.2 项目 4.3 多集群应用 4.4 应用商店 4.5 Ranc ...
【内网福音】如何离线部署Rancher
2019独角兽企业重金招聘Python工程师标准>>> 对于在公司内网环境中.无法访问互联网的用户而言,离线安装部署Rancher是解决问题的关键.本文是Rancher离线部署教程, ...
物联网轻量级开发方案：在K3s部署Shifu，实现云边端闭环
Shifu 是一个 Kubernetes 原生的IoT设备虚拟化框架.Shifu 希望帮助IoT应用开发者以即插即用的方式实现IoT设备的虚拟化.监视.管控和自动化.本文通过在本地集群中部署Shifu ...
k3s 部署nacos-server:v2.1.1-slim
1. 创建 pvc apiVersion: v1 kind: PersistentVolumeClaim metadata:name: nacos-pvcnamespace: tanghao spec ...
Helm部署rancher 高可用集群
Helm部署rancher 高可用集群 Helm简介 Helm是Kubernetes的一个包管理工具,用来简化Kubernetes应用的部署和管理.可以把Helm比作CentOS的yum工具. Hel ...

K3s部署rancher

环境