a36.ansible 生产实战案例 -- 基于kubeadm安装kubernetes v1.23 -- 集群部署(一)
2024-06-04 10:38:03
源码下载地址：https://github.com/raymond999999/kubernetes-ansible
1.高可用Kubernetes集群规划
| 角色 | 机器名 | 机器配置 | ip地址 | 安装软件 |
|---|---|---|---|---|
| ansible | ansible-server.example.local | 2C2G | 172.31.3.100 | ansible |
| master1 | k8s-master01.example.local | 2C4G | 172.31.3.101 | chrony-client、docker、kubeadm 、kubelet、kubectl |
| master2 | k8s-master02.example.local | 2C4G | 172.31.3.102 | chrony-client、docker、kubeadm 、kubelet、kubectl |
| master3 | k8s-master03.example.local | 2C4G | 172.31.3.103 | chrony-client、docker、kubeadm 、kubelet、kubectl |
| ha1 | k8s-ha01.example.local | 2C2G | 172.31.3.104 | chrony-server、haproxy、keepalived |
| ha2 | k8s-ha02.example.local | 2C2G | 172.31.3.105 | chrony-server、haproxy、keepalived |
| harbor1 | k8s-harbor01.example.local | 2C2G | 172.31.3.106 | chrony-client、docker、docker-compose、harbor |
| harbor2 | k8s-harbor02.example.local | 2C2G | 172.31.3.107 | chrony-client、docker、docker-compose、harbor |
| node1 | k8s-node01.example.local | 2C4G | 172.31.3.108 | chrony-client、docker、kubeadm 、kubelet |
| node2 | k8s-node02.example.local | 2C4G | 172.31.3.109 | chrony-client、docker、kubeadm 、kubelet |
| node3 | k8s-node03.example.local | 2C4G | 172.31.3.110 | chrony-client、docker、kubeadm 、kubelet |
软件版本信息和Pod、Service网段规划：
| 配置信息 | 备注 |
|---|---|
| 支持的操作系统版本 | CentOS 7.9/stream 8、Rocky 8、Ubuntu 18.04/20.04 |
| Docker版本 | 20.10.14 |
| Containerd版本 | 1.5.11 |
| kubeadm版本 | 1.23.6 |
| Pod网段 | 192.168.0.0/12 |
| Service网段 | 10.96.0.0/12 |
2.安装ansible和配置
2.1 安装ansible
#CentOS
[root@ansible-server ~]# yum -y install ansible[root@ansible-server ~]# ansible --version
ansible 2.9.25config file = /data/ansible/ansible.cfgconfigured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']ansible python module location = /usr/lib/python2.7/site-packages/ansibleexecutable location = /usr/bin/ansiblepython version = 2.7.5 (default, Oct 14 2020, 14:45:30) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]#ubuntu18.04安装最新版的ansible
root@ubuntu1804:~# apt updateroot@ubuntu1804:~# apt -y install software-properties-commonroot@ubuntu1804:~# apt-add-repository --yes --update ppa:ansible/ansibleroot@ubuntu1804:~# apt -y install ansible
root@ubuntu1804:~# ansible --version
ansible 2.9.27config file = /etc/ansible/ansible.cfgconfigured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']ansible python module location = /usr/lib/python2.7/dist-packages/ansibleexecutable location = /usr/bin/ansiblepython version = 2.7.17 (default, Feb 27 2021, 15:10:58) [GCC 7.5.0]#ubuntu 20.04安装
[root@ubuntu ~]# apt -y install ansible
2.2 配置ansible
[root@ansible-server ~]# mkdir /data/ansible
[root@ansible-server ~]# cd /data/ansible[root@ansible-server ansible]# vim ansible.cfg
[defaults]
inventory = ./inventory
forks = 10
roles_path = ./roles
remote_user = root#下面的IP根据自己的k8s集群主机规划设置
[root@ansible-server ansible]# vim inventory
[master]
172.31.3.101 hname=k8s-master01
172.31.3.102 hname=k8s-master02
172.31.3.103 hname=k8s-master03[ha]
172.31.3.104 hname=k8s-ha01
172.31.3.105 hname=k8s-ha02[harbor]
172.31.3.106 hname=k8s-harbor01
172.31.3.107 hname=k8s-harbor02[node]
172.31.3.108 hname=k8s-node01
172.31.3.109 hname=k8s-node02
172.31.3.110 hname=k8s-node03[all:vars]
domain=example.local[k8s_cluster:children]
master
node[chrony_server:children]
ha[chrony_client:children]
master
node
harbor[keepalives_master]
172.31.3.104[keepalives_backup]
172.31.3.105[haproxy:children]
ha[master01]
172.31.3.101
3.设置客户端网卡名和ip
#rocky8和centos系统设置
[root@172 ~]# bash reset.sh ************************************************************
* 初始化脚本菜单 *
* 1.禁用SELinux 12.修改IP地址和网关地址 *
* 2.关闭防火墙 13.设置主机名 *
* 3.优化SSH 14.设置PS1和系统环境变量 *
* 4.设置系统别名 15.禁用SWAP *
* 5.1-4全设置 16.优化内核参数 *
* 6.设置vimrc配置文件 17.优化资源限制参数 *
* 7.设置软件包仓库 18.Ubuntu设置root用户登录 *
* 8.Minimal安装建议安装软件 19.Ubuntu卸载无用软件包 *
* 9.安装邮件服务并配置邮件 20.重启系统 *
* 10.更改SSH端口号 21.退出 *
* 11.修改网卡名 *
************************************************************请选择相应的编号(1-21): 11
Rocky 8.5 网卡名已修改成功，请重新启动系统后才能生效!************************************************************
* 初始化脚本菜单 *
* 1.禁用SELinux 12.修改IP地址和网关地址 *
* 2.关闭防火墙 13.设置主机名 *
* 3.优化SSH 14.设置PS1和系统环境变量 *
* 4.设置系统别名 15.禁用SWAP *
* 5.1-4全设置 16.优化内核参数 *
* 6.设置vimrc配置文件 17.优化资源限制参数 *
* 7.设置软件包仓库 18.Ubuntu设置root用户登录 *
* 8.Minimal安装建议安装软件 19.Ubuntu卸载无用软件包 *
* 9.安装邮件服务并配置邮件 20.重启系统 *
* 10.更改SSH端口号 21.退出 *
* 11.修改网卡名 *
************************************************************请选择相应的编号(1-21): 12
请输入IP地址:172.31.0.101
IP 172.31.0.101 available!
请输入子网掩码位数:21
请输入网关地址:172.31.0.2
IP 172.31.0.2 available!
Rocky 8.5 IP地址和网关地址已修改成功,请重新启动系统后生效!************************************************************
* 初始化脚本菜单 *
* 1.禁用SELinux 12.修改IP地址和网关地址 *
* 2.关闭防火墙 13.设置主机名 *
* 3.优化SSH 14.设置PS1和系统环境变量 *
* 4.设置系统别名 15.禁用SWAP *
* 5.1-4全设置 16.优化内核参数 *
* 6.设置vimrc配置文件 17.优化资源限制参数 *
* 7.设置软件包仓库 18.Ubuntu设置root用户登录 *
* 8.Minimal安装建议安装软件 19.Ubuntu卸载无用软件包 *
* 9.安装邮件服务并配置邮件 20.重启系统 *
* 10.更改SSH端口号 21.退出 *
* 11.修改网卡名 *
************************************************************请选择相应的编号(1-21): 21#ubuntu系统设置
[C:\~]$ ssh raymond@172.31.7.3Connecting to 172.31.7.3:22...
Connection established.
To escape to local shell, press 'Ctrl+Alt+]'.Welcome to Ubuntu 18.04.6 LTS (GNU/Linux 4.15.0-156-generic x86_64)* Documentation: https://help.ubuntu.com* Management: https://landscape.canonical.com* Support: https://ubuntu.com/advantageSystem information as of Mon Dec 27 13:56:42 CST 2021System load: 0.17 Processes: 193Usage of /: 2.1% of 91.17GB Users logged in: 1Memory usage: 10% IP address for ens33: 172.31.7.3Swap usage: 0%* Super-optimized for small spaces - read how we shrank the memoryfootprint of MicroK8s to make it the smallest full K8s around.https://ubuntu.com/blog/microk8s-memory-optimisation19 updates can be applied immediately.
18 of these updates are standard security updates.
To see these additional updates run: apt list --upgradableNew release '20.04.3 LTS' available.
Run 'do-release-upgrade' to upgrade to it.Last login: Mon Dec 27 13:56:31 2021
/usr/bin/xauth: file /home/raymond/.Xauthority does not exist
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.raymond@ubuntu1804:~$ bash reset.sh ************************************************************
* 初始化脚本菜单 *
* 1.禁用SELinux 12.修改IP地址和网关地址 *
* 2.关闭防火墙 13.设置主机名 *
* 3.优化SSH 14.设置PS1和系统环境变量 *
* 4.设置系统别名 15.禁用SWAP *
* 5.1-4全设置 16.优化内核参数 *
* 6.设置vimrc配置文件 17.优化资源限制参数 *
* 7.设置软件包仓库 18.Ubuntu设置root用户登录 *
* 8.Minimal安装建议安装软件 19.Ubuntu卸载无用软件包 *
* 9.安装邮件服务并配置邮件 20.重启系统 *
* 10.更改SSH端口号 21.退出 *
* 11.修改网卡名 *
************************************************************请选择相应的编号(1-21): 18
请输入密码: 123456
[sudo] password for raymond: Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully
Ubuntu 18.04 root用户登录已设置完成,请重新登录后生效!************************************************************
* 初始化脚本菜单 *
* 1.禁用SELinux 12.修改IP地址和网关地址 *
* 2.关闭防火墙 13.设置主机名 *
* 3.优化SSH 14.设置PS1和系统环境变量 *
* 4.设置系统别名 15.禁用SWAP *
* 5.1-4全设置 16.优化内核参数 *
* 6.设置vimrc配置文件 17.优化资源限制参数 *
* 7.设置软件包仓库 18.Ubuntu设置root用户登录 *
* 8.Minimal安装建议安装软件 19.Ubuntu卸载无用软件包 *
* 9.安装邮件服务并配置邮件 20.重启系统 *
* 10.更改SSH端口号 21.退出 *
* 11.修改网卡名 *
************************************************************请选择相应的编号(1-21): 21
raymond@ubuntu1804:~$ exit
logoutConnection closed.Disconnected from remote host(172.31.7.3:22) at 13:57:16.Type `help' to learn how to use Xshell prompt.[C:\~]$ ssh root@172.31.7.3Connecting to 172.31.7.3:22...
Connection established.
To escape to local shell, press 'Ctrl+Alt+]'.Welcome to Ubuntu 18.04.6 LTS (GNU/Linux 4.15.0-156-generic x86_64)* Documentation: https://help.ubuntu.com* Management: https://landscape.canonical.com* Support: https://ubuntu.com/advantageSystem information as of Mon Dec 27 13:57:47 CST 2021System load: 0.06 Processes: 199Usage of /: 2.1% of 91.17GB Users logged in: 1Memory usage: 11% IP address for ens33: 172.31.7.3Swap usage: 0%* Super-optimized for small spaces - read how we shrank the memoryfootprint of MicroK8s to make it the smallest full K8s around.https://ubuntu.com/blog/microk8s-memory-optimisation19 updates can be applied immediately.
18 of these updates are standard security updates.
To see these additional updates run: apt list --upgradableNew release '20.04.3 LTS' available.
Run 'do-release-upgrade' to upgrade to it.The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law./usr/bin/xauth: file /root/.Xauthority does not exist
root@ubuntu1804:~# mv /home/raymond/reset.sh .
root@ubuntu1804:~# bash reset.sh ************************************************************
* 初始化脚本菜单 *
* 1.禁用SELinux 12.修改IP地址和网关地址 *
* 2.关闭防火墙 13.设置主机名 *
* 3.优化SSH 14.设置PS1和系统环境变量 *
* 4.设置系统别名 15.禁用SWAP *
* 5.1-4全设置 16.优化内核参数 *
* 6.设置vimrc配置文件 17.优化资源限制参数 *
* 7.设置软件包仓库 18.Ubuntu设置root用户登录 *
* 8.Minimal安装建议安装软件 19.Ubuntu卸载无用软件包 *
* 9.安装邮件服务并配置邮件 20.重启系统 *
* 10.更改SSH端口号 21.退出 *
* 11.修改网卡名 *
************************************************************请选择相应的编号(1-21): 11
Ubuntu 18.04 网卡名已修改成功，请重新启动系统后才能生效!************************************************************
* 初始化脚本菜单 *
* 1.禁用SELinux 12.修改IP地址和网关地址 *
* 2.关闭防火墙 13.设置主机名 *
* 3.优化SSH 14.设置PS1和系统环境变量 *
* 4.设置系统别名 15.禁用SWAP *
* 5.1-4全设置 16.优化内核参数 *
* 6.设置vimrc配置文件 17.优化资源限制参数 *
* 7.设置软件包仓库 18.Ubuntu设置root用户登录 *
* 8.Minimal安装建议安装软件 19.Ubuntu卸载无用软件包 *
* 9.安装邮件服务并配置邮件 20.重启系统 *
* 10.更改SSH端口号 21.退出 *
* 11.修改网卡名 *
************************************************************请选择相应的编号(1-21): 12
请输入IP地址:172.31.0.103
IP 172.31.0.103 available!
请输入子网掩码位数:21
请输入网关地址:172.31.0.2
IP 172.31.0.2 available!
Ubuntu 18.04 IP地址和网关地址已修改成功,请重新启动系统后生效!************************************************************
* 初始化脚本菜单 *
* 1.禁用SELinux 12.修改IP地址和网关地址 *
* 2.关闭防火墙 13.设置主机名 *
* 3.优化SSH 14.设置PS1和系统环境变量 *
* 4.设置系统别名 15.禁用SWAP *
* 5.1-4全设置 16.优化内核参数 *
* 6.设置vimrc配置文件 17.优化资源限制参数 *
* 7.设置软件包仓库 18.Ubuntu设置root用户登录 *
* 8.Minimal安装建议安装软件 19.Ubuntu卸载无用软件包 *
* 9.安装邮件服务并配置邮件 20.重启系统 *
* 10.更改SSH端口号 21.退出 *
* 11.修改网卡名 *
************************************************************请选择相应的编号(1-21): 21
4.实现基于key验证的脚本
#下面的IP根据自己的k8s集群主机规划设置
[root@ansible-server ansible]# cat ssh_key.sh
#!/bin/bash
#
#**********************************************************************************************
#Author: Raymond
#QQ: 88563128
#Date: 2021-12-20
#FileName: ssh_key.sh
#URL: raymond.blog.csdn.net
#Description: ssh_key for CentOS 7/8 & Ubuntu 18.04/24.04 & Rocky 8
#Copyright (C): 2021 All rights reserved
#*********************************************************************************************
COLOR="echo -e \\033[01;31m"
END='\033[0m'NET_NAME=`ip addr |awk -F"[: ]" '/^2: e.*/{print $3}'`
IP=`ip addr show ${NET_NAME}| awk -F" +|/" '/global/{print $3}'`
export SSHPASS=123456
HOSTS="
172.31.3.101
172.31.3.102
172.31.3.103
172.31.3.104
172.31.3.105
172.31.3.106
172.31.3.107
172.31.3.108
172.31.3.109
172.31.3.110
172.31.3.188"os(){OS_ID=`sed -rn '/^NAME=/s@.*="([[:alpha:]]+).*"$@\1@p' /etc/os-release`
}ssh_key_push(){rm -f ~/.ssh/id_rsa*ssh-keygen -f /root/.ssh/id_rsa -P '' &> /dev/nullif [ ${OS_ID} == "CentOS" -o ${OS_ID} == "Rocky" ] &> /dev/null;thenrpm -q sshpass &> /dev/null || { ${COLOR}"安装sshpass软件包"${END};yum -y install sshpass &> /dev/null; }elsedpkg -S sshpass &> /dev/null || { ${COLOR}"安装sshpass软件包"${END};apt -y install sshpass &> /dev/null; }fisshpass -e ssh-copy-id -o StrictHostKeyChecking=no ${IP} &> /dev/null[ $? -eq 0 ] && echo ${IP} is finished || echo ${IP} is falsefor i in ${HOSTS};dosshpass -e scp -o StrictHostKeyChecking=no -r /root/.ssh root@${i}: &> /dev/null[ $? -eq 0 ] && echo ${i} is finished || echo ${i} is falsedonefor i in ${HOSTS};doscp /root/.ssh/known_hosts ${i}:.ssh/ &> /dev/null[ $? -eq 0 ] && echo ${i} is finished || echo ${i} is falsedone
}main(){osssh_key_push
}main[root@ansible-server ansible]# bash ssh_key.sh
172.31.3.100 is finished
172.31.3.101 is finished
172.31.3.102 is finished
172.31.3.103 is finished
172.31.3.104 is finished
172.31.3.105 is finished
172.31.3.106 is finished
172.31.3.107 is finished
172.31.3.108 is finished
172.31.3.109 is finished
172.31.3.110 is finished
172.31.3.101 is finished
172.31.3.102 is finished
172.31.3.103 is finished
172.31.3.104 is finished
172.31.3.105 is finished
172.31.3.106 is finished
172.31.3.107 is finished
172.31.3.108 is finished
172.31.3.109 is finished
172.31.3.110 is finished
5.系统初始化和安装软件包
5.1 系统初始化
[root@ansible-server ansible]# mkdir -p roles/reset/{tasks,templates,vars}[root@ansible-server ansible]# cd roles/reset/
[root@ansible-server reset]# ls
tasks templates vars[root@ansible-server reset]# vim templates/yum8.repo.j2
[BaseOS]
name=BaseOS
{% if ansible_distribution =="Rocky" %}
baseurl=https://{{ ROCKY_URL }}/rocky/$releasever/BaseOS/$basearch/os/
{% elif ansible_distribution=="CentOS" %}
baseurl=https://{{ URL }}/centos/$releasever-stream/BaseOS/$basearch/os/
{% endif %}
gpgcheck=1
{% if ansible_distribution =="Rocky" %}
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial
{% elif ansible_distribution=="CentOS" %}
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
{% endif %}[AppStream]
name=AppStream
{% if ansible_distribution =="Rocky" %}
baseurl=https://{{ ROCKY_URL }}/rocky/$releasever/AppStream/$basearch/os/
{% elif ansible_distribution=="CentOS" %}
baseurl=https://{{ URL }}/centos/$releasever-stream/AppStream/$basearch/os/
{% endif %}
gpgcheck=1
{% if ansible_distribution =="Rocky" %}
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial
{% elif ansible_distribution=="CentOS" %}
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
{% endif %}[extras]
name=extras
{% if ansible_distribution =="Rocky" %}
baseurl=https://{{ ROCKY_URL }}/rocky/$releasever/extras/$basearch/os/
{% elif ansible_distribution=="CentOS" %}
baseurl=https://{{ URL }}/centos/$releasever-stream/extras/$basearch/os/
{% endif %}
gpgcheck=1
{% if ansible_distribution =="Rocky" %}
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial
{% elif ansible_distribution=="CentOS" %}
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
{% endif %}{% if ansible_distribution =="Rocky" %}
[plus]
{% elif ansible_distribution=="CentOS" %}
[centosplus]
{% endif %}
{% if ansible_distribution =="Rocky" %}
name=plus
{% elif ansible_distribution=="CentOS" %}
name=centosplus
{% endif %}
{% if ansible_distribution =="Rocky" %}
baseurl=https://{{ ROCKY_URL }}/rocky/$releasever/plus/$basearch/os/
{% elif ansible_distribution=="CentOS" %}
baseurl=https://{{ URL }}/centos/$releasever-stream/centosplus/$basearch/os/
{% endif %}
gpgcheck=1
{% if ansible_distribution =="Rocky" %}
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial
{% elif ansible_distribution=="CentOS" %}
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
{% endif %}[PowerTools]
name=PowerTools
{% if ansible_distribution =="Rocky" %}
baseurl=https://{{ ROCKY_URL }}/rocky/$releasever/PowerTools/$basearch/os/
{% elif ansible_distribution=="CentOS" %}
baseurl=https://{{ URL }}/centos/$releasever-stream/PowerTools/$basearch/os/
{% endif %}
gpgcheck=1
{% if ansible_distribution =="Rocky" %}
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial
{% elif ansible_distribution=="CentOS" %}
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
{% endif %}[epel]
name=epel
{% if ansible_distribution =="Rocky" %}
baseurl=https://{{ ROCKY_URL }}/fedora/epel/$releasever/Everything/$basearch/
{% elif ansible_distribution=="CentOS" %}
baseurl=https://{{ URL }}/epel/$releasever/Everything/$basearch/
{% endif %}
gpgcheck=1
{% if ansible_distribution =="Rocky" %}
gpgkey=https://{{ ROCKY_URL }}/fedora/epel/RPM-GPG-KEY-EPEL-$releasever
{% elif ansible_distribution=="CentOS" %}
gpgkey=https://{{ URL }}/epel/RPM-GPG-KEY-EPEL-$releasever
{% endif %}[root@ansible-server reset]# vim templates/yum7.repo.j2
[base]
name=base
baseurl=https://{{ URL }}/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-$releasever[extras]
name=extras
baseurl=https://{{ URL }}/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-$releasever[updates]
name=updates
baseurl=https://{{ URL }}/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-$releasever[centosplus]
name=centosplus
baseurl=https://{{ URL }}/centos/$releasever/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-$releasever[epel]
name=epel
baseurl=https://{{ URL }}/epel/$releasever/$basearch/
gpgcheck=1
gpgkey=https://{{ URL }}/epel/RPM-GPG-KEY-EPEL-$releasever[root@ansible-server reset]# vim templates/apt.list.j2
deb http://{{ URL }}/ubuntu/ {{ ansible_distribution_release }} main restricted universe multiverse
deb-src http://{{ URL }}/ubuntu/ {{ ansible_distribution_release }} main restricted universe multiversedeb http://{{ URL }}/ubuntu/ {{ ansible_distribution_release }}-security main restricted universe multiverse
deb-src http://{{ URL }}/ubuntu/ {{ ansible_distribution_release }}-security main restricted universe multiversedeb http://{{ URL }}/ubuntu/ {{ ansible_distribution_release }}-updates main restricted universe multiverse
deb-src http://{{ URL }}/ubuntu/ {{ ansible_distribution_release }}-updates main restricted universe multiversedeb http://{{ URL }}/ubuntu/ {{ ansible_distribution_release }}-proposed main restricted universe multiverse
deb-src http://{{ URL }}/ubuntu/ {{ ansible_distribution_release }}-proposed main restricted universe multiversedeb http://{{ URL }}/ubuntu/ {{ ansible_distribution_release }}-backports main restricted universe multiverse
deb-src http://{{ URL }}/ubuntu/ {{ ansible_distribution_release }}-backports main restricted universe multiverse#下面VIP设置成自己的keepalived里的VIP(虚拟IP)地址，HARBOR_DOMAIN的地址设置成自己的harbor域名地址
[root@ansible-server reset]# vim vars/main.yml
VIP: 172.31.3.188
HARBOR_DOMAIN: harbor.raymonds.cc
ROCKY_URL: mirrors.ustc.edu.cn
URL: mirrors.cloud.tencent.com[root@ansible-server reset]# vim tasks/set_hostname.yml
- name: set hostnamehostname:name: "{{ hname }}.{{ domain }}"[root@ansible-server reset]# vim tasks/set_hosts.yml
- name: set hosts filelineinfile:path: "/etc/hosts"line: "{{ item }} {{hostvars[item].ansible_hostname}}.{{ domain }} {{hostvars[item].ansible_hostname}}"loop:"{{ play_hosts }}"
- name: set hosts file2lineinfile:path: "/etc/hosts"line: "{{ item }}"loop:- "{{ VIP }} k8s-lb"- "{{ VIP }} {{ HARBOR_DOMAIN }}"[root@ansible-server reset]# vim tasks/disable_selinux.yml
- name: disable selinuxreplace:path: /etc/sysconfig/selinuxregexp: '^(SELINUX=).*'replace: '\1disabled'when:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")[root@ansible-server reset]# vim tasks/disable_firewall.yml
- name: disable firewallsystemd:name: firewalldstate: stoppedenabled: nowhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")
- name: disable ufwsystemd:name: ufwstate: stoppedenabled: nowhen:- ansible_distribution=="Ubuntu"[root@ansible-server reset]# vim tasks/disable_networkmanager.yml
- name: disable NetworkManagersystemd:name: NetworkManagerstate: stoppedenabled: nowhen:- ansible_distribution=="CentOS"- ansible_distribution_major_version=="7"[root@ansible-server reset]# vim tasks/disable_swap.yml
- name: disable swapreplace:path: /etc/fstabregexp: '^(.*swap.*)'replace: '#\1'
- name: get sd numbershell:cmd: lsblk|awk -F"[ └─]" '/SWAP/{printf $3}'register: SD_NAMEwhen:- ansible_distribution=="Ubuntu"- ansible_distribution_major_version=="20"
- name: disable swap for ubuntu20shell:cmd: systemctl mask dev-{{ SD_NAME.stdout}}.swapwhen:- ansible_distribution=="Ubuntu"- ansible_distribution_major_version=="20"[root@ansible-server reset]# vim tasks/set_limits.yml
- name: set limitshell:cmd: ulimit -SHn 65535
- name: set limits.conf filelineinfile:path: "/etc/security/limits.conf"line: "{{ item }}"loop:- "* soft nofile 655360"- "* hard nofile 131072"- "* soft nproc 655350"- "* hard nproc 655350"- "* soft memlock unlimited"- "* hard memlock unlimited" [root@ansible-server reset]# vim tasks/optimization_sshd.yml
- name: optimization sshd disable UseDNSreplace:path: /etc/ssh/sshd_configregexp: '^#(UseDNS).*'replace: '\1 no'
- name: optimization sshd diaable CentOS or Rocky GSSAPIAuthenticationreplace:path: /etc/ssh/sshd_configregexp: '^(GSSAPIAuthentication).*'replace: '\1 no'when:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")
- name: optimization sshd diaable Ubuntu GSSAPIAuthenticationreplace:path: /etc/ssh/sshd_configregexp: '^#(GSSAPIAuthentication).*'replace: '\1 no'notify:- restart sshdwhen:- ansible_distribution=="Ubuntu"[root@ansible-server reset]# vim tasks/set_alias.yml
- name: set CentOS or Rocky aliaslineinfile:path: ~/.bashrcline: "{{ item }}"loop:- "alias cdnet=\"cd /etc/sysconfig/network-scripts\""- "alias vie0=\"vim /etc/sysconfig/network-scripts/ifcfg-eth0\""- "alias vie1=\"vim /etc/sysconfig/network-scripts/ifcfg-eth1\""- "alias scandisk=\"echo '- - -' > /sys/class/scsi_host/host0/scan;echo '- - -' > /sys/class/scsi_host/host1/scan;echo '- - -' > /sys/class/scsi_host/host2/scan\""when:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")
- name: set Ubuntu aliaslineinfile:path: ~/.bashrcline: "{{ item }}"loop:- "alias cdnet=\"cd /etc/netplan\""- "alias scandisk=\"echo '- - -' > /sys/class/scsi_host/host0/scan;echo '- - -' > /sys/class/scsi_host/host1/scan;echo '- - -' > /sys/class/scsi_host/host2/scan\""when:- ansible_distribution=="Ubuntu"[root@ansible-server reset]# vim tasks/set_mirror.yml
- name: find CentOS or Rocky repo filesfind:paths: /etc/yum.repos.d/patterns: "*.repo"register: FILENAMEwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")
- name: delete CentOS or Rocky repo filesfile:path: "{{ item.path }}"state: absentwith_items: "{{ FILENAME.files }}"when:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")
- name: set CentOS8 or Rocky8 Mirror warehousetemplate:src: yum8.repo.j2dest: /etc/yum.repos.d/base.repowhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")- ansible_distribution_major_version=="8"
- name: set CentOS7 Mirror warehousetemplate:src: yum7.repo.j2dest: /etc/yum.repos.d/base.repowhen:- ansible_distribution=="CentOS"- ansible_distribution_major_version=="7"
- name: set Ubuntu Mirror warehousetemplate:src: apt.list.j2dest: /etc/apt/sources.listwhen:- ansible_distribution=="Ubuntu"
- name: delete lock filesfile:path: "{{ item }}"state: absentloop:- /var/lib/dpkg/lock- /var/lib/apt/lists/lock- /var/cache/apt/archives/lockwhen:- ansible_distribution=="Ubuntu"
- name: apt updateapt:update_cache: yes force: yes when:- ansible_distribution=="Ubuntu"[root@ansible-server reset]# vim tasks/main.yml
- include: set_hostname.yml
- include: set_hosts.yml
- include: disable_selinux.yml
- include: disable_firewall.yml
- include: disable_networkmanager.yml
- include: disable_swap.yml
- include: set_limits.yml
- include: optimization_sshd.yml
- include: set_alias.yml
- include: set_mirror.yml[root@ansible-server reset]# cd ../../
[root@ansible-server ansible]# tree roles/reset/
[root@ansible-server ansible]# tree roles/reset/
roles/reset/
├── tasks
│ ├── disable_firewall.yml
│ ├── disable_networkmanager.yml
│ ├── disable_selinux.yml
│ ├── disable_swap.yml
│ ├── main.yml
│ ├── optimization_sshd.yml
│ ├── set_alias.yml
│ ├── set_hostname.yml
│ ├── set_hosts.yml
│ ├── set_limits.yml
│ └── set_mirror.yml
├── templates
│ ├── apt.list.j2
│ ├── yum7.repo.j2
│ └── yum8.repo.j2
└── vars└── main.yml3 directories, 15 files[root@ansible-server ansible]# vim reset_role.yml
---
- hosts: allroles:- role: reset[root@ansible-server ansible]# ansible-playbook reset_role.yml
5.2 安装软件包
[root@ansible-server ansible]# mkdir -p roles/reset-installpackage/{files,tasks}[root@ansible-server ansible]# cd roles/reset-installpackage/
[root@ansible-server reset-installpackage]# ls
files tasks[root@ansible-server reset-installpackage]# wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm -P files/[root@ansible-server reset-installpackage]# wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm -P files/[root@ansible-server reset-installpackage]# vim files/ge4.18_ipvs.conf
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip[root@ansible-server reset-installpackage]# vim files/lt4.18_ipvs.conf
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack_ipv4
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip[root@ansible-server reset-installpackage]# vim files/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384[root@ansible-server reset-installpackage]# vim tasks/install_package.yml
- name: install Centos or Rocky packageyum:name: vim,tree,lrzsz,wget,jq,psmisc,net-tools,telnet,gitwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")
- name: install Centos8 or Rocky8 packageyum:name: rsyncwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")- ansible_distribution_major_version=="8"
- name: install Ubuntu packageapt:name: tree,lrzsz,jqforce: yes when:- ansible_distribution=="Ubuntu"[root@ansible-server reset-installpackage]# vim tasks/set_centos7_kernel.yml
- name: update CentOS7yum:name: '*'state: latestexclude: kernel*when:- ansible_distribution=="CentOS"- ansible_distribution_major_version=="7"
- name: copy CentOS7 kernel filescopy: src: "{{ item }}"dest: /tmploop:- kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm- kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpmwhen:- ansible_distribution=="CentOS"- ansible_distribution_major_version=="7"
- name: Finding RPM files find: paths: "/tmp" patterns: "*.rpm" register: RPM_RESULTwhen:- ansible_distribution=="CentOS"- ansible_distribution_major_version=="7"
- name: Install RPM yum: name: "{{ item.path }}" with_items: "{{ RPM_RESULT.files }}" when:- ansible_distribution=="CentOS"- ansible_distribution_major_version=="7"
- name: delete kernel filesfile:path: "{{ item.path }}"state: absent with_items: "{{ RPM_RESULT.files }}" when:- ansible_distribution=="CentOS"- ansible_distribution_major_version=="7"
- name: set grubshell:cmd: grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg; grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"when:- ansible_distribution=="CentOS"- ansible_distribution_major_version=="7"[root@ansible-server reset-installpackage]# vim tasks/install_ipvsadm.yml
- name: install CentOS or Rocky ipvsadmyum:name: ipvsadm,ipset,sysstat,conntrack,libseccompwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")- inventory_hostname in groups.k8s_cluster
- name: install Ubuntu ipvsadmapt:name: ipvsadm,ipset,sysstat,conntrack,libseccomp-devforce: yes when:- ansible_distribution=="Ubuntu"- inventory_hostname in groups.k8s_cluster[root@ansible-server reset-installpackage]# vim tasks/set_ipvs.yml
- name: configuration load_modshell:cmd: |modprobe -- ip_vsmodprobe -- ip_vs_rrmodprobe -- ip_vs_wrrmodprobe -- ip_vs_shwhen:- inventory_hostname in groups.k8s_cluster
- name: configuration load_mod kernel ge4.18shell:cmd: modprobe -- nf_conntrackwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky") or (ansible_distribution=="Ubuntu" and ansible_distribution_major_version=="20")- inventory_hostname in groups.k8s_cluster
- name: configuration load_mod kernel lt4.18shell:cmd: modprobe -- nf_conntrack_ipv4when:- (ansible_distribution=="Ubuntu" and ansible_distribution_major_version=="18")- inventory_hostname in groups.k8s_cluster
- name: Copy ge4.18_ipvs.conf filecopy: src: ge4.18_ipvs.confdest: /etc/modules-load.d/ipvs.confwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky") or (ansible_distribution=="Ubuntu" and ansible_distribution_major_version=="20")- inventory_hostname in groups.k8s_cluster
- name: Copy lt4.18_ipvs.conf filecopy: src: lt4.18_ipvs.confdest: /etc/modules-load.d/ipvs.confwhen:- (ansible_distribution=="Ubuntu" and ansible_distribution_major_version=="18")- inventory_hostname in groups.k8s_cluster
- name: start systemd-modules-load service systemd:name: systemd-modules-loadstate: startedenabled: yeswhen:- inventory_hostname in groups.k8s_cluster[root@ansible-server reset-installpackage]# vim tasks/set_k8s_kernel.yml
- name: copy k8s.conf filecopy: src: k8s.confdest: /etc/sysctl.d/
- name: Load kernel configshell:cmd: "sysctl --system"[root@ansible-server reset-installpackage]# vim tasks/reboot_system.yml
- name: reboot systemreboot:[root@ansible-server reset-installpackage]# vim tasks/main.yml
- include: install_package.yml
- include: set_centos7_kernel.yml
- include: install_ipvsadm.yml
- include: set_ipvs.yml
- include: set_k8s_kernel.yml
- include: reboot_system.yml[root@ansible-server reset-installpackage]# cd ../../
[root@ansible-server ansible]# tree roles/reset-installpackage/
roles/reset-installpackage/
├── files
│ ├── ge4.18_ipvs.conf
│ ├── k8s.conf
│ ├── kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm
│ ├── kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm
│ └── lt4.18_ipvs.conf
└── tasks├── install_ipvsadm.yml├── install_package.yml├── main.yml├── reboot_system.yml├── set_centos7_kernel.yml├── set_ipvs.yml└── set_k8s_kernel.yml2 directories, 12 files[root@ansible-server ansible]# vim reset_installpackage_role.yml
---
- hosts: allserial: 3roles:- role: reset-installpackage[root@ansible-server ansible]# ansible-playbook reset_installpackage_role.yml
6.安装chrony
6.1 安装chrony-server
[root@ansible-server ansible]# mkdir -p roles/chrony-server/{tasks,handlers}[root@ansible-server ansible]# cd roles/chrony-server/
[root@ansible-server chrony-server]# ls
handlers tasks[root@ansible-server chrony-server]# vim tasks/install_chrony_yum.yml
- name: install CentOS or Rocky chronyyum:name: chronywhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")
- name: delete CentOS or Rocky /etc/chrony.conf file contains '^pool.*' string linelineinfile:path: /etc/chrony.confregexp: '^pool.*'state: absentwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")notify:- restart chronyd
- name: delete CentOS or Rocky /etc/chrony.conf file contains '^server.*' string linelineinfile:path: /etc/chrony.confregexp: '^server.*'state: absentwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")notify:- restart chronyd
- name: add Time server for CentOS or Rocky /etc/chrony.conf filelineinfile:path: /etc/chrony.confinsertafter: '^# Please consider .*'line: "server ntp.aliyun.com iburst\nserver time1.cloud.tencent.com iburst\nserver ntp.tuna.tsinghua.edu.cn iburst"when:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")notify:- restart chronyd
- name: Substitution '^#(allow).*' string for CentOS or Rocky /etc/chrony.conf filereplace:path: /etc/chrony.confregexp: '^#(allow).*'replace: '\1 0.0.0.0/0'when:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")notify:- restart chronyd
- name: Substitution '^#(local).*' string for CentOS or Rocky /etc/chrony.conf filereplace:path: /etc/chrony.confregexp: '^#(local).*'replace: '\1 stratum 10'when:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")notify:- restart chronyd[root@ansible-server chrony-server]# vim tasks/install_chrony_apt.yml
- name: delete lock filesfile:path: "{{ item }}"state: absentloop:- /var/lib/dpkg/lock- /var/lib/apt/lists/lock- /var/cache/apt/archives/lockwhen:- ansible_distribution=="Ubuntu"
- name: apt updateapt:update_cache: yesforce: yes when:- ansible_distribution=="Ubuntu"
- name: install Ubuntu chronyapt:name: chronyforce: yeswhen:- ansible_distribution=="Ubuntu"
- name: delete Ubuntu /etc/chrony/chrony.conf file contains '^pool.*' string linelineinfile:path: /etc/chrony/chrony.confregexp: '^pool.*'state: absentwhen:- ansible_distribution=="Ubuntu"notify:- restart chronyd
- name: add Time server for Ubuntu /etc/chrony/chrony.conf filelineinfile:path: /etc/chrony/chrony.confinsertafter: '^# See http:.*'line: "server ntp.aliyun.com iburst\nserver time1.cloud.tencent.com iburst\nserver ntp.tuna.tsinghua.edu.cn iburst"when:- ansible_distribution=="Ubuntu"
- name: add 'allow 0.0.0.0/0' string and 'local stratum 10' string for Ubuntu /etc/chrony/chrony.conf filelineinfile:path: /etc/chrony/chrony.confline: "{{ item }}"loop:- "allow 0.0.0.0/0"- "local stratum 10"when:- ansible_distribution=="Ubuntu"notify:- restart chronyd[root@ansible-server chrony-server]# vim tasks/service.yml
- name: start chronydsystemd:name: chronydstate: startedenabled: yes[root@ansible-server chrony-server]# vim tasks/main.yml
- include: install_chrony_yum.yml
- include: install_chrony_apt.yml
- include: service.yml[root@ansible-server chrony-server]# vim handlers/main.yml
- name: restart chronydsystemd:name: chronydstate: restarted[root@ansible-server chrony-server]# cd ../../
[root@ansible-server ansible]# tree roles/chrony-server/
roles/chrony-server/
├── handlers
│ └── main.yml
└── tasks├── install_chrony_apt.yml├── install_chrony_yum.yml├── main.yml└── service.yml2 directories, 5 files[root@ansible-server ansible]# vim chrony_server_role.yml
---
- hosts: chrony_serverroles:- role: chrony-server[root@ansible-server ansible]# ansible-playbook chrony_server_role.yml[root@k8s-ha01 ~]# chronyc sources -nv
210 Number of sources = 3
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^- 203.107.6.88 2 6 37 62 -15ms[ -15ms] +/- 35ms
^* 139.199.215.251 2 6 37 62 -10us[+1488us] +/- 37ms
^? 101.6.6.172 0 7 0 - +0ns[ +0ns] +/- 0ns[root@k8s-ha02 ~]# chronyc sources -nv
210 Number of sources = 3
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* 203.107.6.88 2 6 77 3 -4058us[+2582us] +/- 31ms
^+ 139.199.215.251 2 6 77 2 +6881us[+6881us] +/- 33ms
^? 101.6.6.172 0 7 0 - +0ns[ +0ns] +/- 0ns
6.2 安装chrony-client
[root@ansible-server ansible]# mkdir -p roles/chrony-client/{tasks,handlers,vars}
[root@ansible-server ansible]# cd roles/chrony-client/
[root@ansible-server chrony-client]# ls
handlers tasks vars#下面IP设置成chrony-server的IP地址，SERVER1设置ha1的IP地址，SERVER2设置ha2的IP地址
[root@ansible-server chrony-client]# vim vars/main.yml
SERVER1: 172.31.3.104
SERVER2: 172.31.3.105[root@ansible-server chrony-client]# vim tasks/install_chrony_yum.yml
- name: install CentOS or Rocky chronyyum:name: chronywhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")
- name: delete CentOS or Rocky /etc/chrony.conf file contains '^pool.*' string linelineinfile:path: /etc/chrony.confregexp: '^pool.*'state: absentwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")notify:- restart chronyd
- name: delete CentOS or Rocky /etc/chrony.conf file contains '^server.*' string linelineinfile:path: /etc/chrony.confregexp: '^server.*'state: absentwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")notify:- restart chronyd
- name: add Time server for CentOS or Rocky /etc/chrony.conf filelineinfile:path: /etc/chrony.confinsertafter: '^# Please consider .*'line: "server {{ SERVER1 }} iburst\nserver {{ SERVER2 }} iburst"when:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")notify:- restart chronyd[root@ansible-server chrony-client]# vim tasks/install_chrony_apt.yml
- name: delete lock filesfile:path: "{{ item }}"state: absentloop:- /var/lib/dpkg/lock- /var/lib/apt/lists/lock- /var/cache/apt/archives/lockwhen:- ansible_distribution=="Ubuntu"
- name: apt updateapt:update_cache: yesforce: yes when:- ansible_distribution=="Ubuntu"
- name: install Ubuntu chronyapt:name: chronyforce: yeswhen:- ansible_distribution=="Ubuntu"
- name: delete Ubuntu /etc/chrony/chrony.conf file contains '^pool.*' string linelineinfile:path: /etc/chrony/chrony.confregexp: '^pool.*'state: absentwhen:- ansible_distribution=="Ubuntu"notify:- restart chronyd
- name: add Time server for Ubuntu /etc/chrony/chrony.conf filelineinfile:path: /etc/chrony/chrony.confinsertafter: '^# See http:.*'line: "server {{ SERVER1 }} iburst\nserver {{ SERVER2 }} iburst"when:- ansible_distribution=="Ubuntu"notify:- restart chronyd[root@ansible-server chrony-client]# vim tasks/service.yml
- name: start chronydsystemd:name: chronydstate: startedenabled: yes[root@ansible-server chrony-client]# vim tasks/main.yml
- include: install_chrony_yum.yml
- include: install_chrony_apt.yml
- include: service.yml[root@ansible-server chrony-client]# vim handlers/main.yml
- name: restart chronydsystemd:name: chronydstate: restarted[root@ansible-server chrony-client]# cd ../../
[root@ansible-server ansible]# tree roles/chrony-client/
roles/chrony-client/
├── handlers
│ └── main.yml
├── tasks
│ ├── install_chrony_apt.yml
│ ├── install_chrony_yum.yml
│ ├── main.yml
│ └── service.yml
└── vars└── main.yml3 directories, 6 files[root@ansible-server ansible]# vim chrony_client_role.yml
---
- hosts: chrony_clientserial: 3roles:- role: chrony-client[root@ansible-server ansible]# ansible-playbook chrony_client_role.yml[root@k8s-master01 ~]# chronyc sources -nv
210 Number of sources = 2
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* k8s-ha01 3 6 17 28 -57us[ -29us] +/- 31ms
^+ k8s-ha02 3 6 17 29 +204us[ +231us] +/- 34ms
7.安装haproxy
[root@ansible-server ansible]# mkdir -p roles/haproxy/{tasks,vars,files,templates}
[root@ansible-server ansible]# cd roles/haproxy/
[root@ansible-server haproxy]# ls
files tasks templates vars[root@ansible-server haproxy]# wget http://www.lua.org/ftp/lua-5.4.4.tar.gz -P files/
[root@ansible-server haproxy]# wget https://www.haproxy.org/download/2.4/src/haproxy-2.4.15.tar.gz -P files/[root@ansible-server haproxy]# vim files/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=syslog.target network.target[Service]
ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecStart=/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/lib/haproxy/haproxy.pid
ExecReload=/bin/kill -USR2 $MAINPID[Install]
WantedBy=multi-user.target#下面VIP设置成自己的keepalived里的VIP(虚拟IP)地址
[root@ansible-server haproxy]# vim vars/main.yml
SRC_DIR: /usr/local/src
LUA_FILE: lua-5.4.4.tar.gz
HAPROXY_FILE: haproxy-2.4.15.tar.gz
HAPROXY_INSTALL_DIR: /apps/haproxy
STATS_AUTH_USER: admin
STATS_AUTH_PASSWORD: 123456
VIP: 172.31.3.188[root@ansible-server haproxy]# vim templates/haproxy.cfg.j2
global
maxconn 100000
chroot {{ HAPROXY_INSTALL_DIR }}
stats socket /var/lib/haproxy/haproxy.sock mode 600 level admin
uid 99
gid 99
daemon
pidfile /var/lib/haproxy/haproxy.pid
log 127.0.0.1 local3 infodefaults
option http-keep-alive
option forwardfor
maxconn 100000
mode http
timeout connect 300000ms
timeout client 300000ms
timeout server 300000mslisten statsmode httpbind 0.0.0.0:9999stats enablelog globalstats uri /haproxy-statusstats auth {{ STATS_AUTH_USER }}:{{ STATS_AUTH_PASSWORD }}listen kubernetes-6443bind {{ VIP }}:6443mode tcplog global{% for i in groups.master %}server {{ i }} {{ i }}:6443 check inter 3s fall 2 rise 5{% endfor %}listen harbor-80bind {{ VIP }}:80mode httplog globalbalance source{% for i in groups.harbor %}server {{ i }} {{ i }}:80 check inter 3s fall 2 rise 5{% endfor %}[root@ansible-server haproxy]# vim tasks/install_package.yml
- name: install CentOS or Rocky depend on the packageyum:name: gcc,make,gcc-c++,glibc,glibc-devel,pcre,pcre-devel,openssl,openssl-devel,systemd-devel,libtermcap-devel,ncurses-devel,libevent-devel,readline-develwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")- inventory_hostname in groups.haproxy
- name: delete lock filesfile:path: "{{ item }}"state: absentloop:- /var/lib/dpkg/lock- /var/lib/apt/lists/lock- /var/cache/apt/archives/lockwhen:- ansible_distribution=="Ubuntu"- inventory_hostname in groups.haproxy
- name: apt updateapt:update_cache: yes force: yes when:- ansible_distribution=="Ubuntu"- inventory_hostname in groups.haproxy
- name: install Ubuntu depend on the packageapt:name: gcc,make,openssl,libssl-dev,libpcre3,libpcre3-dev,zlib1g-dev,libreadline-dev,libsystemd-devforce: yes when:- ansible_distribution=="Ubuntu"- inventory_hostname in groups.haproxy[root@ansible-server haproxy]# vim tasks/build_lua.yml
- name: unarchive lua packageunarchive:src: "{{ LUA_FILE }}"dest: "{{ SRC_DIR }}"when:- inventory_hostname in groups.haproxy
- name: get LUA_DIR directoryshell:cmd: echo {{ LUA_FILE }} | sed -nr 's/^(.*[0-9]).([[:lower:]]).*/\1/p'register: LUA_DIRwhen:- inventory_hostname in groups.haproxy
- name: Build and install luashell: chdir: "{{ SRC_DIR }}/{{ LUA_DIR.stdout }}"cmd: make all testwhen:- inventory_hostname in groups.haproxy[root@ansible-server haproxy]# vim tasks/build_haproxy.yml
- name: unarchive haproxy packageunarchive:src: "{{ HAPROXY_FILE }}"dest: "{{ SRC_DIR }}"when:- inventory_hostname in groups.haproxy
- name: get HAPROXY_DIR directoryshell:cmd: echo {{ HAPROXY_FILE }} | sed -nr 's/^(.*[0-9]).([[:lower:]]).*/\1/p'register: HAPROXY_DIRwhen:- inventory_hostname in groups.haproxy
- name: make Haproxyshell: chdir: "{{ SRC_DIR }}/{{ HAPROXY_DIR.stdout }}"cmd: make -j {{ ansible_processor_vcpus }} ARCH=x86_64 TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 USE_CPU_AFFINITY=1 USE_LUA=1 LUA_INC={{ SRC_DIR }}/{{ LUA_DIR.stdout }}/src/ LUA_LIB={{ SRC_DIR }}/{{ LUA_DIR.stdout }}/src/ PREFIX={{ HAPROXY_INSTALL_DIR }}when:- inventory_hostname in groups.haproxy
- name: make install Haproxyshell: chdir: "{{ SRC_DIR }}/{{ HAPROXY_DIR.stdout }}"cmd: make install PREFIX={{ HAPROXY_INSTALL_DIR }}when:- inventory_hostname in groups.haproxy[root@ansible-server haproxy]# vim tasks/config.yml
- name: copy haproxy.service filecopy:src: haproxy.servicedest: /lib/systemd/systemwhen:- inventory_hostname in groups.haproxy
- name: create haproxy linkfile:src: "../..{{ HAPROXY_INSTALL_DIR }}/sbin/{{ item.src }}"dest: "/usr/sbin/{{ item.src }}"state: linkowner: rootgroup: rootmode: 755force: yes with_items:- src: haproxywhen:- inventory_hostname in groups.haproxy
- name: create /etc/haproxy directoryfile:path: /etc/haproxystate: directorywhen:- inventory_hostname in groups.haproxy
- name: create /var/lib/haproxy/ directoryfile:path: /var/lib/haproxy/state: directorywhen:- inventory_hostname in groups.haproxy
- name: copy haproxy.cfg filetemplate:src: haproxy.cfg.j2dest: /etc/haproxy/haproxy.cfgwhen:- inventory_hostname in groups.haproxy
- name: Add the kernelsysctl:name: net.ipv4.ip_nonlocal_bindvalue: "1"when:- inventory_hostname in groups.haproxy
- name: PATH variablecopy:content: 'PATH={{ HAPROXY_INSTALL_DIR }}/sbin:$PATH'dest: /etc/profile.d/haproxy.shwhen:- inventory_hostname in groups.haproxy
- name: PATH variable entryshell:cmd: . /etc/profile.d/haproxy.shwhen:- inventory_hostname in groups.haproxy[root@ansible-server haproxy]# vim tasks/service.yml
- name: start haproxysystemd:name: haproxystate: startedenabled: yesdaemon_reload: yeswhen:- inventory_hostname in groups.haproxy[root@ansible-server haproxy]# vim tasks/main.yml
- include: install_package.yml
- include: build_lua.yml
- include: build_haproxy.yml
- include: config.yml
- include: service.yml[root@ansible-server haproxy]# cd ../../
[root@ansible-server ansible]# tree roles/haproxy/
roles/haproxy/
├── files
│ ├── haproxy-2.4.15.tar.gz
│ ├── haproxy.service
│ └── lua-5.4.4.tar.gz
├── tasks
│ ├── build_haproxy.yml
│ ├── build_lua.yml
│ ├── config.yml
│ ├── install_package.yml
│ ├── main.yml
│ └── service.yml
├── templates
│ └── haproxy.cfg.j2
└── vars└── main.yml4 directories, 11 files[root@ansible-server ansible]# vim haproxy_role.yml
---
- hosts: haproxy:master:harborroles:- role: haproxy[root@ansible-server ansible]# ansible-playbook haproxy_role.yml
8.安装keepalived
8.1 安装keepalived-master
[root@ansible-server ansible]# mkdir -p roles/keepalived-master/{tasks,files,vars,templates}
[root@ansible-server ansible]# cd roles/keepalived-master/
[root@ansible-server keepalived-master]# ls
files tasks templates vars[root@ansible-server keepalived-master]# wget https://keepalived.org/software/keepalived-2.2.7.tar.gz -P files/[root@ansible-server keepalived-master]# vim files/check_haproxy.sh
#!/bin/bash
#
#**********************************************************************************************
#Author: Raymond
#QQ: 88563128
#Date: 2022-01-09
#FileName: check_haproxy.sh
#URL: raymond.blog.csdn.net
#Description: The test script
#Copyright (C): 2022 All rights reserved
#*********************************************************************************************
err=0
for k in $(seq 1 3);docheck_code=$(pgrep haproxy)if [[ $check_code == "" ]]; thenerr=$(expr $err + 1)sleep 1continueelseerr=0breakfi
doneif [[ $err != "0" ]]; thenecho "systemctl stop keepalived"/usr/bin/systemctl stop keepalivedexit 1
elseexit 0
fi#下面VIP设置成自己的keepalived里的VIP(虚拟IP)地址
[root@ansible-server keepalived-master]# vim vars/main.yml
URL: mirrors.cloud.tencent.com
ROCKY_URL: mirrors.sjtug.sjtu.edu.cn
KEEPALIVED_FILE: keepalived-2.2.7.tar.gz
SRC_DIR: /usr/local/src
KEEPALIVED_INSTALL_DIR: /apps/keepalived
STATE: MASTER
PRIORITY: 100
VIP: 172.31.3.188[root@ansible-server keepalived-master]# vim templates/PowerTools.repo.j2
[PowerTools]
name=PowerTools
{% if ansible_distribution =="Rocky" %}
baseurl=https://{{ ROCKY_URL }}/rocky/$releasever/PowerTools/$basearch/os/
{% elif ansible_distribution=="CentOS" %}
baseurl=https://{{ URL }}/centos/$releasever/PowerTools/$basearch/os/
{% endif %}
gpgcheck=1
{% if ansible_distribution =="Rocky" %}
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial
{% elif ansible_distribution=="CentOS" %}
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
{% endif %}[root@ansible-server keepalived-master]# vim templates/keepalived.conf.j2
! Configuration File for keepalivedglobal_defs {router_id LVS_DEVELscript_user rootenable_script_security
}vrrp_script check_haoroxy {script "/etc/keepalived/check_haproxy.sh"interval 5weight -5fall 2 rise 1
}vrrp_instance VI_1 {state {{ STATE }}interface {{ ansible_default_ipv4.interface }}virtual_router_id 51priority {{ PRIORITY }}advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {{{ VIP }} dev {{ ansible_default_ipv4.interface }} label {{ ansible_default_ipv4.interface }}:1}track_script {check_haproxy}
}[root@ansible-server keepalived-master]# vim tasks/install_package.yml
- name: find "[PowerTools]" mirror warehousefind:path: /etc/yum.repos.d/contains: '\[PowerTools\]'register: RETURNwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")- ansible_distribution_major_version=="8"
- name: copy repo filetemplate:src: PowerTools.repo.j2dest: /etc/yum.repos.d/PowerTools.repowhen: - (ansible_distribution=="CentOS" or ansible_distribution=="Rocky") and (ansible_distribution_major_version=="8") - RETURN.matched == 0
- name: install CentOS8 or Rocky8 depend on the packageyum:name: make,gcc,ipvsadm,autoconf,automake,openssl-devel,libnl3-devel,iptables-devel,ipset-devel,file-devel,net-snmp-devel,glib2-devel,pcre2-devel,libnftnl-devel,libmnl-devel,systemd-develwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")- ansible_distribution_major_version=="8"
- name: install CentOS7 depend on the packageyum:name: make,gcc,libnfnetlink-devel,libnfnetlink,ipvsadm,libnl,libnl-devel,libnl3,libnl3-devel,lm_sensors-libs,net-snmp-agent-libs,net-snmp-libs,openssh-server,openssh-clients,openssl,openssl-devel,automake,iproutewhen:- ansible_distribution=="CentOS"- ansible_distribution_major_version=="7"
- name: delete lock filesfile:path: "{{ item }}"state: absentloop:- /var/lib/dpkg/lock- /var/lib/apt/lists/lock- /var/cache/apt/archives/lockwhen:- ansible_distribution=="Ubuntu"
- name: apt updateapt:update_cache: yes force: yes when:- ansible_distribution=="Ubuntu"
- name: install Ubuntu 20.04 depend on the packageapt:name: make,gcc,ipvsadm,build-essential,pkg-config,automake,autoconf,libipset-dev,libnl-3-dev,libnl-genl-3-dev,libssl-dev,libxtables-dev,libip4tc-dev,libip6tc-dev,libipset-dev,libmagic-dev,libsnmp-dev,libglib2.0-dev,libpcre2-dev,libnftnl-dev,libmnl-dev,libsystemd-devforce: yes when:- ansible_distribution=="Ubuntu"- ansible_distribution_major_version=="20"
- name: install Ubuntu 18.04 depend on the packageapt:name: make,gcc,ipvsadm,build-essential,pkg-config,automake,autoconf,iptables-dev,libipset-dev,libnl-3-dev,libnl-genl-3-dev,libssl-dev,libxtables-dev,libip4tc-dev,libip6tc-dev,libipset-dev,libmagic-dev,libsnmp-dev,libglib2.0-dev,libpcre2-dev,libnftnl-dev,libmnl-dev,libsystemd-devforce: yes when:- ansible_distribution=="Ubuntu"- ansible_distribution_major_version=="18"[root@ansible-server keepalived-master]# vim tasks/keepalived_file.yml
- name: unarchive keepalived packageunarchive:src: "{{ KEEPALIVED_FILE }}"dest: "{{ SRC_DIR }}"[root@ansible-server keepalived_master]# vim tasks/build.yml
- name: get KEEPALIVED_DIR directoryshell:cmd: echo {{ KEEPALIVED_FILE }} | sed -nr 's/^(.*[0-9]).([[:lower:]]).*/\1/p'register: KEEPALIVED_DIR
- name: Build and install Keepalivedshell: chdir: "{{ SRC_DIR }}/{{ KEEPALIVED_DIR.stdout }}"cmd: ./configure --prefix={{ KEEPALIVED_INSTALL_DIR }} --disable-fwmark
- name: make && make installshell:chdir: "{{ SRC_DIR }}/{{ KEEPALIVED_DIR.stdout }}"cmd: make -j {{ ansible_processor_vcpus }} && make install[root@ansible-server keepalived-master]# vim tasks/config.yml
- name: create /etc/keepalived directoryfile:path: /etc/keepalivedstate: directory
- name: copy keepalived.conf filetemplate:src: keepalived.conf.j2dest: /etc/keepalived/keepalived.conf
- name: copy check_haproxy.sh filecopy:src: check_haproxy.shdest: /etc/keepalived/mode: 0755
- name: copy keepalived.service filecopy:remote_src: Truesrc: "{{ SRC_DIR }}/{{ KEEPALIVED_DIR.stdout }}/keepalived/keepalived.service"dest: /lib/systemd/system/
- name: PATH variablecopy:content: 'PATH={{ KEEPALIVED_INSTALL_DIR }}/sbin:$PATH'dest: /etc/profile.d/keepalived.sh
- name: PATH variable entryshell:cmd: . /etc/profile.d/keepalived.sh[root@ansible-server keepalived-master]# vim tasks/service.yml
- name: start keepalivedsystemd:name: keepalivedstate: startedenabled: yesdaemon_reload: yes[root@ansible-server keepalived-master]# vim tasks/main.yml
- include: install_package.yml
- include: keepalived_file.yml
- include: build.yml
- include: config.yml
- include: service.yml[root@ansible-server keepalived-master]# cd ../../
[root@ansible-server ansible]# tree roles/keepalived-master/
roles/keepalived-master/
├── files
│ ├── check_haproxy.sh
│ └── keepalived-2.2.7.tar.gz
├── tasks
│ ├── build.yml
│ ├── config.yml
│ ├── install_package.yml
│ ├── keepalived_file.yml
│ ├── main.yml
│ └── service.yml
├── templates
│ ├── keepalived.conf.j2
│ └── PowerTools.repo.j2
└── vars└── main.yml4 directories, 11 files[root@ansible-server ansible]# vim keepalived_master_role.yml
---
- hosts: keepalives_masterroles:- role: keepalived-master[root@ansible-server ansible]# ansible-playbook keepalived_master_role.yml
8.2 安装keepalived-backup
[root@ansible-server ansible]# mkdir -p roles/keepalived-backup/{tasks,files,vars,templates}
[root@ansible-server ansible]# cd roles/keepalived-backup/
[root@ansible-server keepalived-master]# ls
files tasks templates vars[root@ansible-server keepalived-backup]# wget https://keepalived.org/software/keepalived-2.2.7.tar.gz -P files/[root@ansible-server keepalived-backup]# vim files/check_haproxy.sh
#!/bin/bash
#
#**********************************************************************************************
#Author: Raymond
#QQ: 88563128
#Date: 2022-01-09
#FileName: check_haproxy.sh
#URL: raymond.blog.csdn.net
#Description: The test script
#Copyright (C): 2022 All rights reserved
#*********************************************************************************************
err=0
for k in $(seq 1 3);docheck_code=$(pgrep haproxy)if [[ $check_code == "" ]]; thenerr=$(expr $err + 1)sleep 1continueelseerr=0breakfi
doneif [[ $err != "0" ]]; thenecho "systemctl stop keepalived"/usr/bin/systemctl stop keepalivedexit 1
elseexit 0
fi#下面VIP设置成自己的keepalived里的VIP(虚拟IP)地址
[root@ansible-server keepalived-backup]# vim vars/main.yml
URL: mirrors.cloud.tencent.com
ROCKY_URL: mirrors.sjtug.sjtu.edu.cn
KEEPALIVED_FILE: keepalived-2.2.7.tar.gz
SRC_DIR: /usr/local/src
KEEPALIVED_INSTALL_DIR: /apps/keepalived
STATE: BACKUP
PRIORITY: 90
VIP: 172.31.3.188[root@ansible-server keepalived-backup]# vim templates/PowerTools.repo.j2
[PowerTools]
name=PowerTools
{% if ansible_distribution =="Rocky" %}
baseurl=https://{{ ROCKY_URL }}/rocky/$releasever/PowerTools/$basearch/os/
{% elif ansible_distribution=="CentOS" %}
baseurl=https://{{ URL }}/centos/$releasever/PowerTools/$basearch/os/
{% endif %}
gpgcheck=1
{% if ansible_distribution =="Rocky" %}
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial
{% elif ansible_distribution=="CentOS" %}
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
{% endif %}[root@ansible-server keepalived-backup]# vim templates/keepalived.conf.j2
! Configuration File for keepalivedglobal_defs {router_id LVS_DEVELscript_user rootenable_script_security
}vrrp_script check_haoroxy {script "/etc/keepalived/check_haproxy.sh"interval 5weight -5fall 2 rise 1
}vrrp_instance VI_1 {state {{ STATE }}interface {{ ansible_default_ipv4.interface }}virtual_router_id 51priority {{ PRIORITY }}advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {{{ VIP }} dev {{ ansible_default_ipv4.interface }} label {{ ansible_default_ipv4.interface }}:1}track_script {check_haproxy}
}[root@ansible-server keepalived-backup]# vim tasks/install_package.yml
- name: find "[PowerTools]" mirror warehousefind:path: /etc/yum.repos.d/contains: '\[PowerTools\]'register: RETURNwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")- ansible_distribution_major_version=="8"
- name: copy repo filetemplate:src: PowerTools.repo.j2dest: /etc/yum.repos.d/PowerTools.repowhen: - (ansible_distribution=="CentOS" or ansible_distribution=="Rocky") and (ansible_distribution_major_version=="8") - RETURN.matched == 0
- name: install CentOS8 or Rocky8 depend on the packageyum:name: make,gcc,ipvsadm,autoconf,automake,openssl-devel,libnl3-devel,iptables-devel,ipset-devel,file-devel,net-snmp-devel,glib2-devel,pcre2-devel,libnftnl-devel,libmnl-devel,systemd-develwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")- ansible_distribution_major_version=="8"
- name: install CentOS7 depend on the packageyum:name: make,gcc,libnfnetlink-devel,libnfnetlink,ipvsadm,libnl,libnl-devel,libnl3,libnl3-devel,lm_sensors-libs,net-snmp-agent-libs,net-snmp-libs,openssh-server,openssh-clients,openssl,openssl-devel,automake,iproutewhen:- ansible_distribution=="CentOS"- ansible_distribution_major_version=="7"
- name: delete lock filesfile:path: "{{ item }}"state: absentloop:- /var/lib/dpkg/lock- /var/lib/apt/lists/lock- /var/cache/apt/archives/lockwhen:- ansible_distribution=="Ubuntu"
- name: apt updateapt:update_cache: yes force: yes when:- ansible_distribution=="Ubuntu"
- name: install Ubuntu 20.04 depend on the packageapt:name: make,gcc,ipvsadm,build-essential,pkg-config,automake,autoconf,libipset-dev,libnl-3-dev,libnl-genl-3-dev,libssl-dev,libxtables-dev,libip4tc-dev,libip6tc-dev,libipset-dev,libmagic-dev,libsnmp-dev,libglib2.0-dev,libpcre2-dev,libnftnl-dev,libmnl-dev,libsystemd-devforce: yes when:- ansible_distribution=="Ubuntu"- ansible_distribution_major_version=="20"
- name: install Ubuntu 18.04 depend on the packageapt:name: make,gcc,ipvsadm,build-essential,pkg-config,automake,autoconf,iptables-dev,libipset-dev,libnl-3-dev,libnl-genl-3-dev,libssl-dev,libxtables-dev,libip4tc-dev,libip6tc-dev,libipset-dev,libmagic-dev,libsnmp-dev,libglib2.0-dev,libpcre2-dev,libnftnl-dev,libmnl-dev,libsystemd-devforce: yes when:- ansible_distribution=="Ubuntu"- ansible_distribution_major_version=="18"[root@ansible-server keepalived-backup]# vim tasks/keepalived_file.yml
- name: unarchive keepalived packageunarchive:src: "{{ KEEPALIVED_FILE }}"dest: "{{ SRC_DIR }}"[root@ansible-server keepalived_backup]# vim tasks/build.yml
- name: get KEEPALIVED_DIR directoryshell:cmd: echo {{ KEEPALIVED_FILE }} | sed -nr 's/^(.*[0-9]).([[:lower:]]).*/\1/p'register: KEEPALIVED_DIR
- name: Build and install Keepalivedshell: chdir: "{{ SRC_DIR }}/{{ KEEPALIVED_DIR.stdout }}"cmd: ./configure --prefix={{ KEEPALIVED_INSTALL_DIR }} --disable-fwmark
- name: make && make installshell:chdir: "{{ SRC_DIR }}/{{ KEEPALIVED_DIR.stdout }}"cmd: make -j {{ ansible_processor_vcpus }} && make install[root@ansible-server keepalived-backup]# vim tasks/config.yml
- name: create /etc/keepalived directoryfile:path: /etc/keepalivedstate: directory
- name: copy keepalived.conf filetemplate:src: keepalived.conf.j2dest: /etc/keepalived/keepalived.conf
- name: copy check_haproxy.sh filecopy:src: check_haproxy.shdest: /etc/keepalived/mode: 0755
- name: copy keepalived.service filecopy:remote_src: Truesrc: "{{ SRC_DIR }}/{{ KEEPALIVED_DIR.stdout }}/keepalived/keepalived.service"dest: /lib/systemd/system/
- name: PATH variablecopy:content: 'PATH={{ KEEPALIVED_INSTALL_DIR }}/sbin:$PATH'dest: /etc/profile.d/keepalived.sh
- name: PATH variable entryshell:cmd: . /etc/profile.d/keepalived.sh[root@ansible-server keepalived-backup]# vim tasks/service.yml
- name: start keepalivedsystemd:name: keepalivedstate: startedenabled: yesdaemon_reload: yes[root@ansible-server keepalived-backup]# vim tasks/main.yml
- include: install_package.yml
- include: keepalived_file.yml
- include: build.yml
- include: config.yml
- include: service.yml[root@ansible-server keepalived-backup]# cd ../../
roles/keepalived-backup/
├── files
│ ├── check_haproxy.sh
│ └── keepalived-2.2.7.tar.gz
├── tasks
│ ├── build.yml
│ ├── config.yml
│ ├── install_package.yml
│ ├── keepalived_file.yml
│ ├── main.yml
│ └── service.yml
├── templates
│ ├── keepalived.conf.j2
│ └── PowerTools.repo.j2
└── vars└── main.yml4 directories, 11 files[root@ansible-server ansible]# vim keepalived_backup_role.yml
---
- hosts: keepalives_backuproles:- role: keepalived-backup[root@ansible-server ansible]# ansible-playbook keepalived_backup_role.yml
9.安装harbor
9.1 docker基于镜像仓库
[root@ansible-server ansible]# mkdir -p roles/docker/{tasks,vars,templates}[root@ansible-server ansible]# cd roles/docker/
[root@ansible-server docker]# ls
tasks templates vars[root@ansible-server docker]# vim templates/daemon.json.j2
{"registry-mirrors": ["https://registry.docker-cn.com","http://hub-mirror.c.163.com","https://docker.mirrors.ustc.edu.cn"],"insecure-registries": ["{{ HARBOR_DOMAIN }}"],"exec-opts": ["native.cgroupdriver=systemd"],"max-concurrent-downloads": 10,"max-concurrent-uploads": 5,"log-opts": {"max-size": "300m","max-file": "2" },"live-restore": true
}#下面HARBOR_DOMAIN的地址设置成自己的harbor域名地址
[root@ansible-server docker]# vim vars/main.yml
DOCKER_VERSION: 20.10.14
DOCKER_MIRRORS: mirrors.cloud.tencent.com
HARBOR_DOMAIN: harbor.raymonds.cc[root@ansible-server docker]# vim tasks/install_docker_yum.yml
- name: add CentOS or Rocky docker mirror warehouseyum_repository:name: docker-cedescription: docker-cefile: docker-cebaseurl: https://{{ DOCKER_MIRRORS }}/docker-ce/linux/centos/{{ ansible_distribution_major_version }}/x86_64/stable/gpgkey: https://{{ DOCKER_MIRRORS }}/docker-ce/linux/centos/gpg gpgcheck: yeswhen: - (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")
- name: install CentOS or Rocky dockeryum: name: docker-ce-{{ DOCKER_VERSION }},docker-ce-cli-{{ DOCKER_VERSION }}when: - (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")[root@ansible-server docker]# vim tasks/install_docker_apt.yml
- name: delete lock filesfile:path: "{{ item }}"state: absentloop:- /var/lib/dpkg/lock- /var/lib/apt/lists/lock- /var/cache/apt/archives/lockwhen:- ansible_distribution=="Ubuntu"
- name: apt updateapt:update_cache: yes force: yes when:- ansible_distribution=="Ubuntu"
- name: install Ubuntu docker depend on the packageapt:name: apt-transport-https,ca-certificates,curl,software-properties-commonforce: yeswhen:- ansible_distribution=="Ubuntu"
- name: import Ubuntu docker keyapt_key:url: https://{{ DOCKER_MIRRORS }}/docker-ce/linux/ubuntu/gpgwhen:- ansible_distribution=="Ubuntu"
- name: import Ubuntu docker installation sourceapt_repository:repo: "deb [arch=amd64] https://{{ DOCKER_MIRRORS }}/docker-ce/linux/ubuntu {{ ansible_distribution_release }} stable"filename: docker-cewhen:- ansible_distribution=="Ubuntu"
- name: delete /var/lib/dpkg/lock filefile:path: /var/lib/dpkg/lockstate: absentwhen:- ansible_distribution=="Ubuntu"
- name: delete /var/lib/apt/lists/lock filefile:path: /var/lib/apt/lists/lockstate: absentwhen:- ansible_distribution=="Ubuntu"
- name: delete /var/cache/apt/archives/lock filefile:path: /var/cache/apt/archives/lockstate: absentwhen:- ansible_distribution=="Ubuntu"
- name: apt updateapt:update_cache: yesforce: yeswhen:- ansible_distribution=="Ubuntu"
- name: install Ubuntu dockerapt:name: docker-ce=5:{{ DOCKER_VERSION }}~3-0~ubuntu-{{ ansible_distribution_release }},docker-ce-cli=5:{{ DOCKER_VERSION }}~3-0~ubuntu-{{ ansible_distribution_release }}force: yeswhen:- ansible_distribution=="Ubuntu"[root@ansible-server docker]# vim tasks/set_mirror_accelerator.yml
- name: mkdir /etc/dockerfile:path: /etc/dockerstate: directory
- name: set mirror_acceleratortemplate:src: daemon.json.j2dest: /etc/docker/daemon.json[root@ansible-server docker]# vim tasks/service.yml
- name: start dockersystemd:name: dockerstate: startedenabled: yesdaemon_reload: yes[root@ansible-server docker]# vim tasks/set_alias.yml
- name: set docker aliaslineinfile:path: ~/.bashrcline: "{{ item }}"loop:- "alias rmi=\"docker images -qa|xargs docker rmi -f\""- "alias rmc=\"docker ps -qa|xargs docker rm -f\""[root@ansible-server docker]# vim tasks/set_swap.yml
- name: set WARNING No swap limit supportreplace:path: /etc/default/grubregexp: '^(GRUB_CMDLINE_LINUX=.*)\"$'replace: '\1 swapaccount=1"'when:- ansible_distribution=="Ubuntu"
- name: update-grubshell:cmd: update-grubwhen:- ansible_distribution=="Ubuntu"
- name: reboot Ubuntu systemreboot:when:- ansible_distribution=="Ubuntu"[root@ansible-server docker]# vim tasks/main.yml
- include: install_docker_yum.yml
- include: install_docker_apt.yml
- include: set_mirror_accelerator.yml
- include: service.yml
- include: set_alias.yml
- include: set_swap.yml[root@ansible-server docker]# cd ../../
[root@ansible-server ansible]# tree roles/docker
roles/docker
├── tasks
│ ├── install_docker_apt.yml
│ ├── install_docker_yum.yml
│ ├── main.yml
│ ├── service.yml
│ ├── set_alias.yml
│ ├── set_mirror_accelerator.yml
│ └── set_swap.yml
├── templates
│ └── daemon.json.j2
└── vars└── main.yml3 directories, 9 files
9.2 docker-compose
[root@ansible-server ansible]# mkdir -p roles/docker-compose/{tasks,files}
[root@ansible-server ansible]# cd roles/docker-compose/
[root@ansible-server docker-compose]# ls
files tasks[root@ansible-server docker-compose]# wget https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64 -P files[root@ansible-server docker-compose]# vim tasks/install_docker_compose.yml
- name: copy docker compose filecopy:src: docker-compose-linux-x86_64dest: /usr/bin/docker-composemode: 755[root@ansible-server docker-compose]# vim tasks/main.yml
- include: install_docker_compose.yml[root@ansible-server ansible]# tree roles/docker-compose/
roles/docker-compose/
├── files
│ └── docker-compose-linux-x86_64
└── tasks├── install_docker_compose.yml└── main.yml2 directories, 3 files
9.3 harbor
[root@ansible-server ansible]# mkdir -p roles/harbor/{tasks,files,templates,vars,meta}[root@ansible-server ansible]# cd roles/harbor/
[root@ansible-server harbor]# ls
files meta tasks templates vars[root@ansible-server harbor]# wget https://github.com/goharbor/harbor/releases/download/v2.5.0/harbor-offline-installer-v2.5.0.tgz -P files/[root@ansible-server harbor]# vim templates/harbor.service.j2
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/vmware/harbor[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/bin/docker-compose -f {{ HARBOR_INSTALL_DIR }}/harbor/docker-compose.yml up
ExecStop=/usr/bin/docker-compose -f {{ HARBOR_INSTALL_DIR }}/harbor/docker-compose.yml down[Install]
WantedBy=multi-user.target[root@ansible-server harbor]# vim vars/main.yml
HARBOR_INSTALL_DIR: /apps
HARBOR_VERSION: 2.5.0
HARBOR_ADMIN_PASSWORD: 123456[root@ansible-server harbor]# vim tasks/harbor_files.yml
- name: create HARBOR_INSTALL_DIR directoryfile:path: "{{ HARBOR_INSTALL_DIR }}"state: directory
- name: unarchive harbor packageunarchive:src: "harbor-offline-installer-v{{ HARBOR_VERSION }}.tgz"dest: "{{ HARBOR_INSTALL_DIR }}/"creates: "{{ HARBOR_INSTALL_DIR }}/harbor"[root@ansible-server harbor]# vim tasks/config.yml
- name: mv harbor.ymlshell: cmd: mv {{ HARBOR_INSTALL_DIR }}/harbor/harbor.yml.tmpl {{ HARBOR_INSTALL_DIR }}/harbor/harbor.ymlcreates: "{{ HARBOR_INSTALL_DIR }}/harbor/harbor.yml"
- name: set harbor.yml file 'hostname' string linereplace: path: "{{ HARBOR_INSTALL_DIR }}/harbor/harbor.yml"regexp: '^(hostname:) .*'replace: '\1 {{ ansible_default_ipv4.address }}'
- name: set harbor.yml file 'harbor_admin_password' string linereplace: path: "{{ HARBOR_INSTALL_DIR }}/harbor/harbor.yml"regexp: '^(harbor_admin_password:) .*'replace: '\1 {{ HARBOR_ADMIN_PASSWORD }}'
- name: set harbor.yml file 'https' string linereplace:path: "{{ HARBOR_INSTALL_DIR }}/harbor/harbor.yml"regexp: '^(https:)'replace: '#\1'
- name: set harbor.yml file 'port' string linereplace: path: "{{ HARBOR_INSTALL_DIR }}/harbor/harbor.yml"regexp: ' (port: 443)'replace: '# \1'
- name: set harbor.yml file 'certificate' string linereplace: path: "{{ HARBOR_INSTALL_DIR }}/harbor/harbor.yml"regexp: ' (certificate: .*)'replace: '# \1'
- name: set harbor.yml file 'private_key' string linereplace: path: "{{ HARBOR_INSTALL_DIR }}/harbor/harbor.yml"regexp: ' (private_key: .*)'replace: '# \1'[root@ansible-server harbor]# vim tasks/install_python.yml
- name: install CentOS or Rocky pythonyum:name: python3when:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")
- name: delete lock filesfile:path: "{{ item }}"state: absentloop:- /var/lib/dpkg/lock- /var/lib/apt/lists/lock- /var/cache/apt/archives/lockwhen:- ansible_distribution=="Ubuntu"
- name: apt updateapt:update_cache: yes force: yes when:- ansible_distribution=="Ubuntu"
- name: install Ubuntu pythonapt:name: python3when:- ansible_distribution=="Ubuntu"[root@ansible-server harbor]# vim tasks/install_harbor.yml
- name: install harborshell:cmd: "{{ HARBOR_INSTALL_DIR }}/harbor/install.sh"[root@ansible-server harbor]# vim tasks/service_file.yml
- name: copy harbor.servicetemplate:src: harbor.service.j2dest: /lib/systemd/system/harbor.service[root@ansible-server harbor]# vim tasks/service.yml
- name: service enablesystemd:name: harborstate: startedenabled: yesdaemon_reload: yes[root@ansible-server harbor]# vim tasks/main.yml
- include: harbor_files.yml
- include: config.yml
- include: install_python.yml
- include: install_harbor.yml
- include: service_file.yml
- include: service.yml#这里是harbor依赖的角色，docker-binary就是docker基于二进制安装,根据情况修改
[root@ansible-server harbor]# vim meta/main.yml
dependencies:- role: docker- role: docker-compose[root@ansible-server harbor]# cd ../../
[root@ansible-server ansible]# tree roles/harbor/
roles/harbor/
├── files
│ └── harbor-offline-installer-v2.5.0.tgz
├── meta
│ └── main.yml
├── tasks
│ ├── config.yml
│ ├── harbor_files.yml
│ ├── install_harbor.yml
│ ├── install_python.yml
│ ├── main.yml
│ ├── service_file.yml
│ └── service.yml
├── templates
│ └── harbor.service.j2
└── vars└── main.yml5 directories, 11 files[root@ansible-server ansible]# vim harbor_role.yml
---
- hosts: harborroles:- role: harbor[root@ansible-server ansible]# ansible-playbook harbor_role.yml
9.4 创建harbor仓库
这步一定要做，不然后面镜像下载了上传不到harbor，ansible会执行出错
在harbor01新建项目google_containers
在harbor02新建项目google_containers
在harbor02上新建目标
在harbor02上新建规则
在harbor01上新建目标
在harbor01上新建规则
10.安装containerd
[root@ansible-server ansible]# mkdir -p roles/containerd/{vars,tasks}
[root@ansible-server ansible]# cd roles/containerd/
[root@ansible-server containerd]# ls
tasks vars[root@ansible-server containerd]# vim vars/main.yml
DOCKER_VERSION: 20.10.14
DOCKER_MIRRORS: mirrors.cloud.tencent.com
HARBOR_DOMAIN: harbor.raymonds.cc
USERNAME: admin
PASSWORD: 123456[root@ansible-server containerd]# vim tasks/install_docker_yum.yml
- name: add CentOS or Rocky docker mirror warehouseyum_repository:name: docker-cedescription: docker-cefile: docker-cebaseurl: https://{{ DOCKER_MIRRORS }}/docker-ce/linux/centos/{{ ansible_distribution_major_version }}/x86_64/stable/gpgkey: https://{{ DOCKER_MIRRORS }}/docker-ce/linux/centos/gpg gpgcheck: yeswhen: - (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")
- name: install CentOS or Rocky dockeryum: name: docker-ce-{{ DOCKER_VERSION }},docker-ce-cli-{{ DOCKER_VERSION }}when: - (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")[root@ansible-server containerd]# vim tasks/install_docker_apt.yml
- name: delete lock filesfile:path: "{{ item }}"state: absentloop:- /var/lib/dpkg/lock- /var/lib/apt/lists/lock- /var/cache/apt/archives/lockwhen:- ansible_distribution=="Ubuntu"
- name: apt updateapt:update_cache: yes force: yes when:- ansible_distribution=="Ubuntu"
- name: install Ubuntu docker depend on the packageapt:name: apt-transport-https,ca-certificates,curl,software-properties-commonforce: yeswhen:- ansible_distribution=="Ubuntu"
- name: import Ubuntu docker keyapt_key:url: https://{{ DOCKER_MIRRORS }}/docker-ce/linux/ubuntu/gpgwhen:- ansible_distribution=="Ubuntu"
- name: import Ubuntu docker installation sourceapt_repository:repo: "deb [arch=amd64] https://{{ DOCKER_MIRRORS }}/docker-ce/linux/ubuntu {{ ansible_distribution_release }} stable"filename: docker-cewhen:- ansible_distribution=="Ubuntu"
- name: delete /var/lib/dpkg/lock filefile:path: /var/lib/dpkg/lockstate: absentwhen:- ansible_distribution=="Ubuntu"
- name: delete /var/lib/apt/lists/lock filefile:path: /var/lib/apt/lists/lockstate: absentwhen:- ansible_distribution=="Ubuntu"
- name: delete /var/cache/apt/archives/lock filefile:path: /var/cache/apt/archives/lockstate: absentwhen:- ansible_distribution=="Ubuntu"
- name: apt updateapt:update_cache: yesforce: yeswhen:- ansible_distribution=="Ubuntu"
- name: install Ubuntu dockerapt:name: docker-ce=5:{{ DOCKER_VERSION }}~3-0~ubuntu-{{ ansible_distribution_release }},docker-ce-cli=5:{{ DOCKER_VERSION }}~3-0~ubuntu-{{ ansible_distribution_release }}force: yeswhen:- ansible_distribution=="Ubuntu"[root@ansible-server containerd]# vim tasks/config_containerd.yml
- name: load modules and kernelshell:cmd: |cat > /etc/modules-load.d/containerd.conf <<-EOFoverlaybr_netfilterEOFmodprobe -- overlaymodprobe -- br_netfiltercat > /etc/sysctl.d/99-kubernetes-cri.conf <<-EOFnet.bridge.bridge-nf-call-iptables = 1net.ipv4.ip_forward = 1net.bridge.bridge-nf-call-ip6tables = 1EOFsysctl --system
- name: mkdir /etc/containerdfile:path: /etc/containerdstate: directory
- name: set Containerd config fileshell:cmd: containerd config default | tee /etc/containerd/config.toml
- name: set SystemdCgroup linereplace:path: /etc/containerd/config.tomlregexp: '(.*SystemdCgroup = ).*'replace: '\1true'
- name: set sandbox_image linereplace:path: /etc/containerd/config.tomlregexp: '(.*sandbox_image = ).*'replace: '\1"registry.aliyuncs.com/google_containers/pause:3.6"'
- name: set Mirror Acceleratorlineinfile:path: /etc/containerd/config.tomlinsertafter: '.*registry.mirrors.*'line: " [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"docker.io\"]\n endpoint = [\"https://registry.docker-cn.com\" ,\"http://hub-mirror.c.163.com\" ,\"https://docker.mirrors.ustc.edu.cn\"]\n [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"{{ HARBOR_DOMAIN }}\"]\n endpoint = [\"http://{{ HARBOR_DOMAIN }}\"]"
- name: set Private warehouse certificationlineinfile:path: /etc/containerd/config.tomlinsertafter: '.*registry.configs.*'line: " [plugins.\"io.containerd.grpc.v1.cri\".registry.configs.\"{{ HARBOR_DOMAIN }}\".tls]\n insecure_skip_verify = true\n [plugins.\"io.containerd.grpc.v1.cri\".registry.configs.\"{{ HARBOR_DOMAIN }}\".auth]\n username = \"{{ USERNAME }}\"\n password = \"{{ PASSWORD }}\""[root@ansible-server containerd]# vim tasks/service.yml
- name: start containerdsystemd:name: containerdstate: startedenabled: yesdaemon_reload: yes[root@ansible-server containerd]# vim tasks/set_crictl.yml
- name: set crictl.yamlshell:cmd: |cat > /etc/crictl.yaml <<-EOFruntime-endpoint: unix:///run/containerd/containerd.sockimage-endpoint: unix:///run/containerd/containerd.socktimeout: 10debug: falseEOF[root@ansible-server containerd]# vim tasks/set_alias.yml
- name: set containerd aliaslineinfile:path: ~/.bashrcline: "{{ item }}"loop:- "alias rmi=\"ctr images list -q|xargs ctr images rm\""[root@ansible-server containerd]# vim tasks/main.yml
- include: install_docker_yum.yml
- include: install_docker_apt.yml
- include: config_containerd.yml
- include: service.yml
- include: set_crictl.yml
- include: set_alias.yml[root@ansible-server containerd]# cd ../../
[root@ansible-server ansible]# tree roles/containerd/
roles/containerd/
├── tasks
│ ├── config_containerd.yml
│ ├── install_docker_apt.yml
│ ├── install_docker_yum.yml
│ ├── main.yml
│ ├── service.yml
│ ├── set_alias.yml
│ └── set_crictl.yml
└── vars└── main.yml2 directories, 8 files[root@ansible-server ansible]# vim containerd_role.yml
---
- hosts: k8s_clusterroles:- role: containerd[root@ansible-server ansible]# ansible-playbook containerd_role.yml
11.安装kubeadm
[root@ansible-server ansible]# mkdir -p roles/kubeadm/{templates,vars,tasks}
[root@ansible-server ansible]# cd roles/kubeadm/
[root@ansible-server kubeadm]# ls
files tasks vars[root@ansible-server kubeadm]# vim templates/kubernetes.repo.j2
[kubernetes]
name=Kubernetes
baseurl=https://{{ KUBEADM_MIRRORS }}/kubernetes/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://{{ KUBEADM_MIRRORS }}/kubernetes/yum/doc/yum-key.gpg https://{{ KUBEADM_MIRRORS }}/kubernetes/yum/doc/rpm-package-key.gpg[root@ansible-server kubeadm]# vim templates/kubeadm-config.yaml.j2
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:- system:bootstrappers:kubeadm:default-node-tokentoken: 7t2weq.bjbawausm0jaxuryttl: 24h0m0susages:- signing- authentication
kind: InitConfiguration
localAPIEndpoint:advertiseAddress: {{ ansible_default_ipv4.address }}bindPort: 6443
nodeRegistration:criSocket: /run/containerd/containerd.sockname: k8s-master01.example.localtaints:- effect: NoSchedulekey: node-role.kubernetes.io/master
---
apiServer:certSANs:- {{ VIP }}timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: {{ VIP }}:6443
controllerManager: {}
dns:type: CoreDNS
etcd:local:dataDir: /var/lib/etcd
imageRepository: {{ HARBOR_DOMAIN }}/google_containers
kind: ClusterConfiguration
kubernetesVersion: v{{ KUBEADM_VERSION }}
networking:dnsDomain: {{ domain }}podSubnet: {{ POD_SUBNET }}serviceSubnet: {{ SERVICE_SUBNET }}
scheduler: {}#下面HARBOR_DOMAIN的地址设置成自己的harbor域名地址，VIP设置成自己的keepalived里的VIP(虚拟IP)地址，POD_SUBNET改成自己规划的容器网段，SERVICE_SUBNET改成自己规划的service网段地址
[root@ansible-server kubeadm]# vim vars/main.yml
KUBEADM_MIRRORS: mirrors.aliyun.com
KUBEADM_VERSION: 1.23.6
HARBOR_DOMAIN: harbor.raymonds.cc
USERNAME: admin
PASSWORD: 123456
VIP: 172.31.3.188
POD_SUBNET: 192.168.0.0/12
SERVICE_SUBNET: 10.96.0.0/12[root@ansible-server kubeadm]# vim tasks/install_kubeadm_yum.yml
- name: set CentOS or Rocky kubernetes mirror warehousetemplate:src: kubernetes.repo.j2dest: /etc/yum.repos.d/kubernetes.repowhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")
- name: install CentOS or Rocky kubeadm for masteryum:name: kubelet-{{ KUBEADM_VERSION }},kubeadm-{{ KUBEADM_VERSION }},kubectl-{{ KUBEADM_VERSION }}when:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")- inventory_hostname in groups.master
- name: install CentOS or Rocky kubeadm for nodeyum:name: kubelet-{{ KUBEADM_VERSION }},kubeadm-{{ KUBEADM_VERSION }}when:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")- inventory_hostname in groups.node[root@ansible-server kubeadm]# vim tasks/install_kubeadm_apt.yml
- name: delete lock filesfile:path: "{{ item }}"state: absentloop:- /var/lib/dpkg/lock- /var/lib/apt/lists/lock- /var/cache/apt/archives/lockwhen:- ansible_distribution=="Ubuntu"
- name: apt updateapt:update_cache: yes force: yes when:- ansible_distribution=="Ubuntu"
- name: install Ubuntu kubernetes depend on the packageapt:name: apt-transport-httpsforce: yeswhen:- ansible_distribution=="Ubuntu"
- name: import Ubuntu kubernetes keyapt_key:url: https://{{ KUBEADM_MIRRORS }}/kubernetes/apt/doc/apt-key.gpgwhen:- ansible_distribution=="Ubuntu"
- name: import Ubuntu kubernetes installation sourceapt_repository:repo: "deb https://{{ KUBEADM_MIRRORS }}/kubernetes/apt kubernetes-xenial main"filename: kuberneteswhen:- ansible_distribution=="Ubuntu"
- name: delete lock filesfile:path: "{{ item }}"state: absentloop:- /var/lib/dpkg/lock- /var/lib/apt/lists/lock- /var/cache/apt/archives/lockwhen:- ansible_distribution=="Ubuntu"
- name: apt updateapt:update_cache: yesforce: yeswhen:- ansible_distribution=="Ubuntu"
- name: install Ubuntu kubeadm for masterapt:name: kubelet={{ KUBEADM_VERSION }}-00,kubeadm={{ KUBEADM_VERSION }}-00,kubectl={{ KUBEADM_VERSION }}-00force: yeswhen:- ansible_distribution=="Ubuntu"- inventory_hostname in groups.master
- name: install Ubuntu kubeadm for nodeapt:name: kubelet={{ KUBEADM_VERSION }}-00,kubeadm={{ KUBEADM_VERSION }}-00force: yeswhen:- ansible_distribution=="Ubuntu"- inventory_hostname in groups.node[root@ansible-server kubeadm]# vim tasks/set_kubelet.yml
- name: set kubelet configshell:cmd: |cat > /etc/sysconfig/kubelet <<EOFKUBELET_KUBEADM_ARGS="--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"EOF[root@ansible-server kubeadm]# vim tasks/service.yml
- name: start kubeletsystemd:name: kubeletstate: startedenabled: yesdaemon_reload: yes[root@ansible-server kubeadm]# vim tasks/download_images.yml
- name: get kubeadm versionshell:cmd: kubeadm config images list --kubernetes-version=v{{ KUBEADM_VERSION }} | awk -F "/" '{print $NF}'register: KUBEADM_IMAGES_VERSIONwhen:- ansible_hostname=="k8s-master01"
- name: install CentOS or Rocky expect packageyum:name: expectwhen:- (ansible_distribution=="CentOS" or ansible_distribution=="Rocky")- ansible_hostname=="k8s-master01"
- name: install Ubuntu expect packageapt:name: expectforce: yeswhen:- ansible_distribution=="Ubuntu"- ansible_hostname=="k8s-master01"
- name: download kubeadm imageshell: cmd: |{% for i in KUBEADM_IMAGES_VERSION.stdout_lines %}ctr images pull --all-platforms registry.aliyuncs.com/google_containers/{{ i }}ctr images tag registry.aliyuncs.com/google_containers/{{ i }} {{ HARBOR_DOMAIN }}/google_containers/{{ i }}ctr images remove registry.aliyuncs.com/google_containers/{{ i }}expect <<EOFspawn ctr images push --plain-http -u {{ USERNAME }} {{ HARBOR_DOMAIN }}/google_containers/{{ i }}expect "Password:" { send "{{ PASSWORD }}\n";exp_continue }EOF{% endfor %}when:- ansible_hostname=="k8s-master01"[root@ansible-server kubeadm]# vim tasks/copy_kubeadm_config.yml
- name: copy kubeadm_config.yml filetemplate:src: kubeadm-config.yaml.j2dest: /root/kubeadm-config.yamlwhen:- ansible_hostname=="k8s-master01"[root@ansible-server kubeadm]# vim tasks/main.yml
- include: install_kubeadm_yum.yml
- include: install_kubeadm_apt.yml
- include: set_kubelet.yml
- include: service.yml
- include: download_images.yml
- include: copy_kubeadm_config.yml[root@ansible-server kubeadm]# cd ../../
[root@ansible-server ansible]# tree roles/kubeadm
roles/kubeadm
├── tasks
│ ├── copy_kubeadm_config.yml
│ ├── download_images.yml
│ ├── install_kubeadm_apt.yml
│ ├── install_kubeadm_yum.yml
│ ├── main.yml
│ ├── service.yml
│ └── set_kubelet.yml
├── templates
│ ├── kubeadm-config.yaml.j2
│ └── kubernetes.repo.j2
└── vars└── main.yml3 directories, 10 files[root@ansible-server ansible]# vim kubeadm_role.yml
---
- hosts: master:noderoles:- role: kubeadm[root@ansible-server ansible]# ansible-playbook kubeadm_role.yml
12.集群初始化
[root@k8s-master01 ~]# kubeadm init --config /root/kubeadm-config.yaml --upload-certs
[init] Using Kubernetes version: v1.23.6
[preflight] Running pre-flight checks[WARNING FileExisting-tc]: tc not found in system path
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master01.example.local kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.example.local] and IPs [10.96.0.1 172.31.3.101 172.31.3.188]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master01.example.local localhost] and IPs [172.31.3.101 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master01.example.local localhost] and IPs [172.31.3.101 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 22.035806 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster
NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently.
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
a356a0307e08f530b8b93b5d1b6c3c8d372f8808be369632aa61434996e5928b
[mark-control-plane] Marking the node k8s-master01.example.local as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-master01.example.local as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 7t2weq.bjbawausm0jaxury
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxyYour Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/configAlternatively, if you are the root user, you can run:export KUBECONFIG=/etc/kubernetes/admin.confYou should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:https://kubernetes.io/docs/concepts/cluster-administration/addons/You can now join any number of the control-plane node running the following command on each as root:kubeadm join 172.31.3.188:6443 --token 7t2weq.bjbawausm0jaxury \--discovery-token-ca-cert-hash sha256:fd3bcda7669c8e047de6d663f12c4a7c662ae9c18bd2edadcb30ab5bf3c0bdcf \--control-plane --certificate-key a356a0307e08f530b8b93b5d1b6c3c8d372f8808be369632aa61434996e5928bPlease note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.Then you can join any number of worker nodes by running the following on each as root:kubeadm join 172.31.3.188:6443 --token 7t2weq.bjbawausm0jaxury \--discovery-token-ca-cert-hash sha256:fd3bcda7669c8e047de6d663f12c4a7c662ae9c18bd2edadcb30ab5bf3c0bdcf
Master01节点配置环境变量，用于访问Kubernetes集群：
[root@k8s-master01 ~]# cat >> /root/.bashrc <<EOF
export KUBECONFIG=/etc/kubernetes/admin.conf
EOF[root@k8s-master01 ~]# source .bashrc #Kubectl 自动补全
#CentOS
[root@k8s-master01 ~]# yum -y install bash-completion#Ubuntu
[root@k8s-master01 ~]# apt -y install bash-completion[root@k8s-master01 ~]# source <(kubectl completion bash)
[root@k8s-master01 ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc
root@k8s-master01:~# exit
logout
查看节点状态：
[root@k8s-master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01.example.local NotReady control-plane,master 2m5s v1.23.6
采用初始化安装方式，所有的系统组件均以容器的方式运行并且在kube-system命名空间内，此时可以查看Pod状态：
[root@k8s-master01 ~]# kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-68db94d5b7-dsb48 0/1 Pending 0 2m38s <none> <none> <none> <none>
coredns-68db94d5b7-xkxcb 0/1 Pending 0 2m38s <none> <none> <none> <none>
etcd-k8s-master01.example.local 1/1 Running 0 2m33s 172.31.3.101 k8s-master01.example.local <none> <none>
kube-apiserver-k8s-master01.example.local 1/1 Running 0 2m33s 172.31.3.101 k8s-master01.example.local <none> <none>
kube-controller-manager-k8s-master01.example.local 1/1 Running 0 2m33s 172.31.3.101 k8s-master01.example.local <none> <none>
kube-proxy-rx8nz 1/1 Running 0 2m38s 172.31.3.101 k8s-master01.example.local <none> <none>
kube-scheduler-k8s-master01.example.local 1/1 Running 0 2m33s 172.31.3.101 k8s-master01.example.local <none> <none>
13.高可用Master
添加master02和master03:
kubeadm join 172.31.3.188:6443 --token 7t2weq.bjbawausm0jaxury \--discovery-token-ca-cert-hash sha256:fd3bcda7669c8e047de6d663f12c4a7c662ae9c18bd2edadcb30ab5bf3c0bdcf \--control-plane --certificate-key a356a0307e08f530b8b93b5d1b6c3c8d372f8808be369632aa61434996e5928b[root@k8s-master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01.example.local NotReady control-plane,master 4m32s v1.23.6
k8s-master02.example.local NotReady control-plane,master 52s v1.23.6
k8s-master03.example.local NotReady control-plane,master 8s v1.23.6
14.Node节点的配置
添加node01、node02、node03:
kubeadm join 172.31.3.188:6443 --token 7t2weq.bjbawausm0jaxury \--discovery-token-ca-cert-hash sha256:fd3bcda7669c8e047de6d663f12c4a7c662ae9c18bd2edadcb30ab5bf3c0bdcf[root@k8s-master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01.example.local NotReady control-plane,master 6m1s v1.23.6
k8s-master02.example.local NotReady control-plane,master 2m21s v1.23.6
k8s-master03.example.local NotReady control-plane,master 97s v1.23.6
k8s-node01.example.local NotReady <none> 33s v1.23.6
k8s-node02.example.local NotReady <none> 18s v1.23.6
k8s-node03.example.local NotReady <none> 5s v1.23.6
15.安装Calico
15.1 安装calico
[root@ansible-server ansible]# mkdir -p roles/calico/{tasks,vars,templates}
[root@ansible-server ansible]# cd roles/calico
[root@ansible-server calico]# ls
tasks templates vars#下面HARBOR_DOMAIN的地址设置成自己的harbor域名地址
[root@ansible-server calico]# vim vars/main.yml
HARBOR_DOMAIN: harbor.raymonds.cc
USERNAME: admin
PASSWORD: 123456[root@ansible-server calico]# cat templates/calico-etcd.yaml.j2
---
# Source: calico/templates/calico-etcd-secrets.yaml
# The following contains k8s Secrets for use with a TLS enabled etcd cluster.
# For information on populating Secrets, see http://kubernetes.io/docs/user-guide/secrets/
apiVersion: v1
kind: Secret
type: Opaque
metadata:name: calico-etcd-secretsnamespace: kube-system
data:# Populate the following with etcd TLS configuration if desired, but leave blank if# not using TLS for etcd.# The keys below should be uncommented and the values populated with the base64# encoded contents of each file that would be associated with the TLS data.# Example command for encoding a file contents: cat <file> | base64 -w 0# etcd-key: null# etcd-cert: null# etcd-ca: null
---
# Source: calico/templates/calico-config.yaml
# This ConfigMap is used to configure a self-hosted Calico installation.
kind: ConfigMap
apiVersion: v1
metadata:name: calico-confignamespace: kube-system
data:# Configure this with the location of your etcd cluster.etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"# If you're using TLS enabled etcd uncomment the following.# You must also populate the Secret below with these files.etcd_ca: "" # "/calico-secrets/etcd-ca"etcd_cert: "" # "/calico-secrets/etcd-cert"etcd_key: "" # "/calico-secrets/etcd-key"# Typha is disabled.typha_service_name: "none"# Configure the backend to use.calico_backend: "bird"# Configure the MTU to use for workload interfaces and tunnels.# By default, MTU is auto-detected, and explicitly setting this field should not be required.# You can override auto-detection by providing a non-zero value.veth_mtu: "0"# The CNI network configuration to install on each node. The special# values in this config will be automatically populated.cni_network_config: |-{"name": "k8s-pod-network","cniVersion": "0.3.1","plugins": [{"type": "calico","log_level": "info","log_file_path": "/var/log/calico/cni/cni.log","etcd_endpoints": "__ETCD_ENDPOINTS__","etcd_key_file": "__ETCD_KEY_FILE__","etcd_cert_file": "__ETCD_CERT_FILE__","etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__","mtu": __CNI_MTU__,"ipam": {"type": "calico-ipam"},"policy": {"type": "k8s"},"kubernetes": {"kubeconfig": "__KUBECONFIG_FILEPATH__"}},{"type": "portmap","snat": true,"capabilities": {"portMappings": true}},{"type": "bandwidth","capabilities": {"bandwidth": true}}]}---
# Source: calico/templates/calico-kube-controllers-rbac.yaml# Include a clusterrole for the kube-controllers component,
# and bind it to the calico-kube-controllers serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: calico-kube-controllers
rules:# Pods are monitored for changing labels.# The node controller monitors Kubernetes nodes.# Namespace and serviceaccount labels are used for policy.- apiGroups: [""]resources:- pods- nodes- namespaces- serviceaccountsverbs:- watch- list- get# Watch for changes to Kubernetes NetworkPolicies.- apiGroups: ["networking.k8s.io"]resources:- networkpoliciesverbs:- watch- list
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: calico-kube-controllers
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: calico-kube-controllers
subjects:
- kind: ServiceAccountname: calico-kube-controllersnamespace: kube-system
------
# Source: calico/templates/calico-node-rbac.yaml
# Include a clusterrole for the calico-node DaemonSet,
# and bind it to the calico-node serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: calico-node
rules:# The CNI plugin needs to get pods, nodes, and namespaces.- apiGroups: [""]resources:- pods- nodes- namespacesverbs:- get# EndpointSlices are used for Service-based network policy rule# enforcement.- apiGroups: ["discovery.k8s.io"]resources:- endpointslicesverbs:- watch - list- apiGroups: [""]resources:- endpoints- servicesverbs:# Used to discover service IPs for advertisement.- watch- list# Pod CIDR auto-detection on kubeadm needs access to config maps.- apiGroups: [""]resources:- configmapsverbs:- get- apiGroups: [""]resources:- nodes/statusverbs:# Needed for clearing NodeNetworkUnavailable flag.- patch---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: calico-node
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: calico-node
subjects:
- kind: ServiceAccountname: calico-nodenamespace: kube-system---
# Source: calico/templates/calico-node.yaml
# This manifest installs the calico-node container, as well
# as the CNI plugins and network config on
# each master and worker node in a Kubernetes cluster.
kind: DaemonSet
apiVersion: apps/v1
metadata:name: calico-nodenamespace: kube-systemlabels:k8s-app: calico-node
spec:selector:matchLabels:k8s-app: calico-nodeupdateStrategy:type: RollingUpdaterollingUpdate:maxUnavailable: 1template:metadata:labels:k8s-app: calico-nodespec:nodeSelector:kubernetes.io/os: linuxhostNetwork: truetolerations:# Make sure calico-node gets scheduled on all nodes.- effect: NoScheduleoperator: Exists# Mark the pod as a critical add-on for rescheduling.- key: CriticalAddonsOnlyoperator: Exists- effect: NoExecuteoperator: ExistsserviceAccountName: calico-node# Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force# deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.terminationGracePeriodSeconds: 0priorityClassName: system-node-criticalinitContainers:# This container installs the CNI binaries# and CNI network config file on each node.- name: install-cniimage: docker.io/calico/cni:v3.21.4command: ["/opt/cni/bin/install"]envFrom:- configMapRef:# Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.name: kubernetes-services-endpointoptional: trueenv:# Name of the CNI config file to create.- name: CNI_CONF_NAMEvalue: "10-calico.conflist"# The CNI network config to install on each node.- name: CNI_NETWORK_CONFIGvalueFrom:configMapKeyRef:name: calico-configkey: cni_network_config# The location of the etcd cluster.- name: ETCD_ENDPOINTSvalueFrom:configMapKeyRef:name: calico-configkey: etcd_endpoints# CNI MTU Config variable- name: CNI_MTUvalueFrom:configMapKeyRef:name: calico-configkey: veth_mtu# Prevents the container from sleeping forever.- name: SLEEPvalue: "false"volumeMounts:- mountPath: /host/opt/cni/binname: cni-bin-dir- mountPath: /host/etc/cni/net.dname: cni-net-dir- mountPath: /calico-secretsname: etcd-certssecurityContext:privileged: true# Adds a Flex Volume Driver that creates a per-pod Unix Domain Socket to allow Dikastes# to communicate with Felix over the Policy Sync API.- name: flexvol-driverimage: docker.io/calico/pod2daemon-flexvol:v3.21.4volumeMounts:- name: flexvol-driver-hostmountPath: /host/driversecurityContext:privileged: truecontainers:# Runs calico-node container on each Kubernetes node. This# container programs network policy and routes on each# host.- name: calico-nodeimage: docker.io/calico/node:v3.21.4envFrom:- configMapRef:# Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.name: kubernetes-services-endpointoptional: trueenv:# The location of the etcd cluster.- name: ETCD_ENDPOINTSvalueFrom:configMapKeyRef:name: calico-configkey: etcd_endpoints# Location of the CA certificate for etcd.- name: ETCD_CA_CERT_FILEvalueFrom:configMapKeyRef:name: calico-configkey: etcd_ca# Location of the client key for etcd.- name: ETCD_KEY_FILEvalueFrom:configMapKeyRef:name: calico-configkey: etcd_key# Location of the client certificate for etcd.- name: ETCD_CERT_FILEvalueFrom:configMapKeyRef:name: calico-configkey: etcd_cert# Set noderef for node controller.- name: CALICO_K8S_NODE_REFvalueFrom:fieldRef:fieldPath: spec.nodeName# Choose the backend to use.- name: CALICO_NETWORKING_BACKENDvalueFrom:configMapKeyRef:name: calico-configkey: calico_backend# Cluster type to identify the deployment type- name: CLUSTER_TYPEvalue: "k8s,bgp"# Auto-detect the BGP IP address.- name: IPvalue: "autodetect"# Enable IPIP- name: CALICO_IPV4POOL_IPIPvalue: "Always"# Enable or Disable VXLAN on the default IP pool.- name: CALICO_IPV4POOL_VXLANvalue: "Never"# Set MTU for tunnel device used if ipip is enabled- name: FELIX_IPINIPMTUvalueFrom:configMapKeyRef:name: calico-configkey: veth_mtu# Set MTU for the VXLAN tunnel device.- name: FELIX_VXLANMTUvalueFrom:configMapKeyRef:name: calico-configkey: veth_mtu# Set MTU for the Wireguard tunnel device.- name: FELIX_WIREGUARDMTUvalueFrom:configMapKeyRef:name: calico-configkey: veth_mtu# The default IPv4 pool to create on startup if none exists. Pod IPs will be# chosen from this range. Changing this value after installation will have# no effect. This should fall within `--cluster-cidr`.# - name: CALICO_IPV4POOL_CIDR# value: "192.168.0.0/16"# Disable file logging so `kubectl logs` works.- name: CALICO_DISABLE_FILE_LOGGINGvalue: "true"# Set Felix endpoint to host default action to ACCEPT.- name: FELIX_DEFAULTENDPOINTTOHOSTACTIONvalue: "ACCEPT"# Disable IPv6 on Kubernetes.- name: FELIX_IPV6SUPPORTvalue: "false"- name: FELIX_HEALTHENABLEDvalue: "true"securityContext:privileged: trueresources:requests:cpu: 250mlifecycle:preStop:exec:command:- /bin/calico-node- -shutdownlivenessProbe:exec:command:- /bin/calico-node- -felix-live- -bird-liveperiodSeconds: 10initialDelaySeconds: 10failureThreshold: 6timeoutSeconds: 10readinessProbe:exec:command:- /bin/calico-node- -felix-ready- -bird-readyperiodSeconds: 10timeoutSeconds: 10volumeMounts:# For maintaining CNI plugin API credentials.- mountPath: /host/etc/cni/net.dname: cni-net-dirreadOnly: false- mountPath: /lib/modulesname: lib-modulesreadOnly: true- mountPath: /run/xtables.lockname: xtables-lockreadOnly: false- mountPath: /var/run/caliconame: var-run-calicoreadOnly: false- mountPath: /var/lib/caliconame: var-lib-calicoreadOnly: false- mountPath: /calico-secretsname: etcd-certs- name: policysyncmountPath: /var/run/nodeagent# For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the# parent directory.- name: sysfsmountPath: /sys/fs/# Bidirectional means that, if we mount the BPF filesystem at /sys/fs/bpf it will propagate to the host.# If the host is known to mount that filesystem already then Bidirectional can be omitted.mountPropagation: Bidirectional- name: cni-log-dirmountPath: /var/log/calico/cnireadOnly: truevolumes:# Used by calico-node.- name: lib-moduleshostPath:path: /lib/modules- name: var-run-calicohostPath:path: /var/run/calico- name: var-lib-calicohostPath:path: /var/lib/calico- name: xtables-lockhostPath:path: /run/xtables.locktype: FileOrCreate- name: sysfshostPath:path: /sys/fs/type: DirectoryOrCreate# Used to install CNI.- name: cni-bin-dirhostPath:path: /opt/cni/bin- name: cni-net-dirhostPath:path: /etc/cni/net.d# Used to access CNI logs.- name: cni-log-dirhostPath:path: /var/log/calico/cni# Mount in the etcd TLS secrets with mode 400.# See https://kubernetes.io/docs/concepts/configuration/secret/- name: etcd-certssecret:secretName: calico-etcd-secretsdefaultMode: 0400# Used to create per-pod Unix Domain Sockets- name: policysynchostPath:type: DirectoryOrCreatepath: /var/run/nodeagent# Used to install Flex Volume Driver- name: flexvol-driver-hosthostPath:type: DirectoryOrCreatepath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
---apiVersion: v1
kind: ServiceAccount
metadata:name: calico-nodenamespace: kube-system---
# Source: calico/templates/calico-kube-controllers.yaml
# See https://github.com/projectcalico/kube-controllers
apiVersion: apps/v1
kind: Deployment
metadata:name: calico-kube-controllersnamespace: kube-systemlabels:k8s-app: calico-kube-controllers
spec:# The controllers can only have a single active instance.replicas: 1selector:matchLabels:k8s-app: calico-kube-controllersstrategy:type: Recreatetemplate:metadata:name: calico-kube-controllersnamespace: kube-systemlabels:k8s-app: calico-kube-controllersspec:nodeSelector:kubernetes.io/os: linuxtolerations:# Mark the pod as a critical add-on for rescheduling.- key: CriticalAddonsOnlyoperator: Exists- key: node-role.kubernetes.io/mastereffect: NoScheduleserviceAccountName: calico-kube-controllerspriorityClassName: system-cluster-critical# The controllers must run in the host network namespace so that# it isn't governed by policy that would prevent it from working.hostNetwork: truecontainers:- name: calico-kube-controllersimage: docker.io/calico/kube-controllers:v3.21.4env:# The location of the etcd cluster.- name: ETCD_ENDPOINTSvalueFrom:configMapKeyRef:name: calico-configkey: etcd_endpoints# Location of the CA certificate for etcd.- name: ETCD_CA_CERT_FILEvalueFrom:configMapKeyRef:name: calico-configkey: etcd_ca# Location of the client key for etcd.- name: ETCD_KEY_FILEvalueFrom:configMapKeyRef:name: calico-configkey: etcd_key# Location of the client certificate for etcd.- name: ETCD_CERT_FILEvalueFrom:configMapKeyRef:name: calico-configkey: etcd_cert# Choose which controllers to run.- name: ENABLED_CONTROLLERSvalue: policy,namespace,serviceaccount,workloadendpoint,nodevolumeMounts:# Mount in the etcd TLS secrets.- mountPath: /calico-secretsname: etcd-certslivenessProbe:exec:command:- /usr/bin/check-status- -lperiodSeconds: 10initialDelaySeconds: 10failureThreshold: 6timeoutSeconds: 10readinessProbe:exec:command:- /usr/bin/check-status- -rperiodSeconds: 10volumes:# Mount in the etcd TLS secrets with mode 400.# See https://kubernetes.io/docs/concepts/configuration/secret/- name: etcd-certssecret:secretName: calico-etcd-secretsdefaultMode: 0440---apiVersion: v1
kind: ServiceAccount
metadata:name: calico-kube-controllersnamespace: kube-system---# This manifest creates a Pod Disruption Budget for Controller to allow K8s Cluster Autoscaler to evictapiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:name: calico-kube-controllersnamespace: kube-systemlabels:k8s-app: calico-kube-controllers
spec:maxUnavailable: 1selector:matchLabels:k8s-app: calico-kube-controllers---
# Source: calico/templates/calico-typha.yaml---
# Source: calico/templates/configure-canal.yaml---
# Source: calico/templates/kdd-crds.yaml#修改下面内容
[root@ansible-server calico]# grep "etcd_endpoints:.*" templates/calico-etcd.yaml.j2 etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"[root@ansible-server calico]# sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "{% for i in groups.master %}https://{{ hostvars[i].ansible_default_ipv4.address }}:2379{% if not loop.last %},{% endif %}{% endfor %}"#g' templates/calico-etcd.yaml.j2 [root@ansible-server calico]# grep "etcd_endpoints:.*" templates/calico-etcd.yaml.j2etcd_endpoints: {% for i in groups.master %}https://{{ hostvars[i].ansible_default_ipv4.address }}{% if not loop.last %},{% endif %}{% endfor %} [root@ansible-server calico]# vim tasks/calico_file.yml
- name: copy calico-etcd.yaml filetemplate:src: calico-etcd.yaml.j2dest: /root/calico-etcd.yamlwhen:- ansible_hostname=="k8s-master01"[root@ansible-server calico]# vim tasks/config.yml
- name: get ETCD_KEY keyshell:cmd: cat /etc/kubernetes/pki/etcd/server.key | base64 | tr -d '\n'register: ETCD_KEYwhen:- ansible_hostname=="k8s-master01"
- name: Modify the ".*etcd-key:.*" linereplace:path: /root/calico-etcd.yamlregexp: '# (etcd-key:) null'replace: '\1 {{ ETCD_KEY.stdout }}'when:- ansible_hostname=="k8s-master01"
- name: get ETCD_CERT keyshell:cmd: cat /etc/kubernetes/pki/etcd/server.crt | base64 | tr -d '\n'register: ETCD_CERTwhen:- ansible_hostname=="k8s-master01"
- name: Modify the ".*etcd-cert:.*" linereplace:path: /root/calico-etcd.yamlregexp: '# (etcd-cert:) null'replace: '\1 {{ ETCD_CERT.stdout }}'when:- ansible_hostname=="k8s-master01"
- name: get ETCD_CA keyshell:cmd: cat /etc/kubernetes/pki/etcd/ca.crt | base64 | tr -d '\n'when:- ansible_hostname=="k8s-master01"register: ETCD_CA
- name: Modify the ".*etcd-ca:.*" linereplace:path: /root/calico-etcd.yamlregexp: '# (etcd-ca:) null'replace: '\1 {{ ETCD_CA.stdout }}'when:- ansible_hostname=="k8s-master01"
- name: Modify the ".*etcd_ca:.*" linereplace:path: /root/calico-etcd.yamlregexp: '(etcd_ca:) ""'replace: '\1 "/calico-secrets/etcd-ca"'when:- ansible_hostname=="k8s-master01"
- name: Modify the ".*etcd_cert:.*" linereplace:path: /root/calico-etcd.yamlregexp: '(etcd_cert:) ""'replace: '\1 "/calico-secrets/etcd-cert"'when:- ansible_hostname=="k8s-master01"
- name: Modify the ".*etcd_key:.*" linereplace:path: /root/calico-etcd.yamlregexp: '(etcd_key:) ""'replace: '\1 "/calico-secrets/etcd-key"'when:- ansible_hostname=="k8s-master01"
- name: Modify the ".*CALICO_IPV4POOL_CIDR.*" linereplace:path: /root/calico-etcd.yamlregexp: '# (- name: CALICO_IPV4POOL_CIDR)'replace: '\1'when:- ansible_hostname=="k8s-master01"
- name: get POD_SUBNETshell:cmd: cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep cluster-cidr= | awk -F= '{print $NF}'register: POD_SUBNETwhen:- ansible_hostname=="k8s-master01"
- name: Modify the ".*192.168.0.0.*" linereplace:path: /root/calico-etcd.yamlregexp: '# (value:) "192.168.0.0/16"'replace: ' \1 "{{ POD_SUBNET.stdout }}"'when:- ansible_hostname=="k8s-master01"
- name: Modify the "image:" linereplace:path: /root/calico-etcd.yamlregexp: '(.*image:) docker.io/calico(/.*)'replace: '\1 {{ HARBOR_DOMAIN }}/google_containers\2'when:- ansible_hostname=="k8s-master01"[root@ansible-server calico]# vim tasks/download_images.yml
- name: get calico versionshell:chdir: /rootcmd: awk -F "/" '/image:/{print $NF}' calico-etcd.yamlregister: CALICO_VERSIONwhen:- ansible_hostname=="k8s-master01"
- name: download calico imageshell:cmd: |{% for i in CALICO_VERSION.stdout_lines %}ctr images pull --all-platforms registry.cn-beijing.aliyuncs.com/raymond9/{{ i }}ctr images tag registry.cn-beijing.aliyuncs.com/raymond9/{{ i }} {{ HARBOR_DOMAIN }}/google_containers/{{ i }}ctr images remove registry.cn-beijing.aliyuncs.com/raymond9/{{ i }}expect <<EOFspawn ctr images push --plain-http -u {{ USERNAME }} {{ HARBOR_DOMAIN }}/google_containers/{{ i }}expect "Password:" { send "{{ PASSWORD }}\n";exp_continue }EOF{% endfor %}when:- ansible_hostname=="k8s-master01"[root@ansible-server calico]# vim tasks/install_calico.yml
- name: install calicoshell:chdir: /rootcmd: "kubectl --kubeconfig=/etc/kubernetes/admin.conf apply -f calico-etcd.yaml"when:- ansible_hostname=="k8s-master01"[root@ansible-server calico]# vim tasks/main.yml
- include: calico_file.yml
- include: config.yml
- include: download_images.yml
- include: install_calico.yml[root@ansible-server calico]# cd ../../
[root@ansible-server ansible]# tree roles/calico
roles/calico
├── tasks
│ ├── calico_file.yml
│ ├── config.yml
│ ├── download_images.yml
│ ├── install_calico.yml
│ └── main.yml
├── templates
│ └── calico-etcd.yaml.j2
└── vars└── main.yml3 directories, 7 files[root@ansible-server ansible]# vim calico_role.yml
---
- hosts: masterroles:- role: calico[root@ansible-server ansible]# ansible-playbook calico_role.yml
15.2 验证calico
[root@k8s-master01 ~]# kubectl get pod -A |grep calico
kube-system calico-kube-controllers-786596c789-prcz4 1/1 Running 0 40s
kube-system calico-node-cpxg4 1/1 Running 0 41s
kube-system calico-node-j5n25 1/1 Running 0 41s
kube-system calico-node-pq4xz 1/1 Running 0 41s
kube-system calico-node-sjgk2 1/1 Running 0 41s
kube-system calico-node-xbkc7 1/1 Running 0 41s
kube-system calico-node-zrww5 1/1 Running 0 41s[root@k8s-master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01.example.local Ready control-plane,master 22m v1.23.6
k8s-master02.example.local Ready control-plane,master 19m v1.23.6
k8s-master03.example.local Ready control-plane,master 18m v1.23.6
k8s-node01.example.local Ready <none> 17m v1.23.6
k8s-node02.example.local Ready <none> 17m v1.23.6
k8s-node03.example.local Ready <none> 16m v1.23.6
16.安装Metrics
16.1 安装metrics
[root@ansible-server ansible]# mkdir -p roles/metrics/{files,vars,tasks}
[root@ansible-server ansible]# cd roles/metrics/
[root@ansible-server metrics]# ls
files tasks vars#下面NODE变量改成自己NODE的IP地址，HARBOR_DOMAIN的地址设置成自己的harbor域名地址
[root@ansible-server metrics]# vim vars/main.yml
NODE: - 172.31.3.108- 172.31.3.109- 172.31.3.110HARBOR_DOMAIN: harbor.raymonds.cc
USERNAME: admin
PASSWORD: 123456[root@ansible-server metrics]# cat files/components.yaml
apiVersion: v1
kind: ServiceAccount
metadata:labels:k8s-app: metrics-servername: metrics-servernamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:k8s-app: metrics-serverrbac.authorization.k8s.io/aggregate-to-admin: "true"rbac.authorization.k8s.io/aggregate-to-edit: "true"rbac.authorization.k8s.io/aggregate-to-view: "true"name: system:aggregated-metrics-reader
rules:
- apiGroups:- metrics.k8s.ioresources:- pods- nodesverbs:- get- list- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:k8s-app: metrics-servername: system:metrics-server
rules:
- apiGroups:- ""resources:- pods- nodes- nodes/stats- namespaces- configmapsverbs:- get- list- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:labels:k8s-app: metrics-servername: metrics-server-auth-readernamespace: kube-system
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccountname: metrics-servernamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:k8s-app: metrics-servername: metrics-server:system:auth-delegator
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:auth-delegator
subjects:
- kind: ServiceAccountname: metrics-servernamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:k8s-app: metrics-servername: system:metrics-server
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:metrics-server
subjects:
- kind: ServiceAccountname: metrics-servernamespace: kube-system
---
apiVersion: v1
kind: Service
metadata:labels:k8s-app: metrics-servername: metrics-servernamespace: kube-system
spec:ports:- name: httpsport: 443protocol: TCPtargetPort: httpsselector:k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:labels:k8s-app: metrics-servername: metrics-servernamespace: kube-system
spec:selector:matchLabels:k8s-app: metrics-serverstrategy:rollingUpdate:maxUnavailable: 0template:metadata:labels:k8s-app: metrics-serverspec:containers:- args:- --cert-dir=/tmp- --secure-port=4443- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname- --kubelet-use-node-status-port- --metric-resolution=15simage: k8s.gcr.io/metrics-server/metrics-server:v0.5.2imagePullPolicy: IfNotPresentlivenessProbe:failureThreshold: 3httpGet:path: /livezport: httpsscheme: HTTPSperiodSeconds: 10name: metrics-serverports:- containerPort: 4443name: httpsprotocol: TCPreadinessProbe:failureThreshold: 3httpGet:path: /readyzport: httpsscheme: HTTPSinitialDelaySeconds: 20periodSeconds: 10resources:requests:cpu: 100mmemory: 200MisecurityContext:readOnlyRootFilesystem: truerunAsNonRoot: truerunAsUser: 1000volumeMounts:- mountPath: /tmpname: tmp-dirnodeSelector:kubernetes.io/os: linuxpriorityClassName: system-cluster-criticalserviceAccountName: metrics-servervolumes:- emptyDir: {}name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:labels:k8s-app: metrics-servername: v1beta1.metrics.k8s.io
spec:group: metrics.k8s.iogroupPriorityMinimum: 100insecureSkipTLSVerify: trueservice:name: metrics-servernamespace: kube-systemversion: v1beta1versionPriority: 100[root@ansible-server metrics]# vim files/components.yaml
...spec:containers:- args:- --cert-dir=/tmp- --secure-port=4443- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname- --kubelet-use-node-status-port- --metric-resolution=15s
#添加下面内容- --kubelet-insecure-tls- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt- --requestheader-username-headers=X-Remote-User- --requestheader-group-headers=X-Remote-Group- --requestheader-extra-headers-prefix=X-Remote-Extra-
...volumeMounts:- mountPath: /tmpname: tmp-dir
#添加下面内容- name: ca-sslmountPath: /etc/kubernetes/pki
...volumes:- emptyDir: {}name: tmp-dir
#添加下面内容- name: ca-sslhostPath:path: /etc/kubernetes/pki
...[root@ansible-server metrics]# vim tasks/metrics_file.yml
- name: copy components.yaml filecopy:src: components.yamldest: /root/components.yaml[root@ansible-server metrics]# vim tasks/config.yml
- name: transfer front-proxy-ca.crt file from mater01 to nodesynchronize:src: /etc/kubernetes/pki/front-proxy-ca.crtdest: /etc/kubernetes/pki/front-proxy-ca.crtmode: pulldelegate_to: "{{ item }}"loop: "{{ NODE }}"
- name: Modify the "image:" linereplace:path: /root/components.yamlregexp: '(.*image:) k8s.gcr.io/metrics-server(/.*)'replace: '\1 {{ HARBOR_DOMAIN }}/google_containers\2'[root@ansible-server metrics]# vim tasks/download_images.yml
- name: get metrics versionshell:chdir: /rootcmd: awk -F "/" '/image:/{print $NF}' components.yamlregister: METRICS_VERSION
- name: download metrics imageshell:cmd: |{% for i in METRICS_VERSION.stdout_lines %}ctr images pull --all-platforms registry.aliyuncs.com/google_containers/{{ i }}ctr images tag registry.aliyuncs.com/google_containers/{{ i }} {{ HARBOR_DOMAIN }}/google_containers/{{ i }}ctr images remove registry.aliyuncs.com/google_containers/{{ i }}expect <<EOFspawn ctr images push --plain-http -u {{ USERNAME }} {{ HARBOR_DOMAIN }}/google_containers/{{ i }}expect "Password:" { send "{{ PASSWORD }}\n";exp_continue }EOF{% endfor %}[root@ansible-server metrics]# vim tasks/install_metrics.yml
- name: install metricsshell:chdir: /rootcmd: "kubectl --kubeconfig=/etc/kubernetes/admin.conf apply -f components.yaml"[root@ansible-server metrics]# vim tasks/main.yml
- include: metrics_file.yml
- include: config.yml
- include: download_images.yml
- include: install_metrics.yml[root@ansible-server metrics]# cd ../../
[root@ansible-server ansible]# tree roles/metrics/
roles/metrics/
├── files
│ └── components.yaml
├── tasks
│ ├── config.yml
│ ├── download_images.yml
│ ├── install_metrics.yml
│ ├── main.yml
│ └── metrics_file.yml
└── vars└── main.yml3 directories, 7 files[root@ansible-server ansible]# vim metrics_role.yml
---
- hosts: master01roles:- role: metrics[root@ansible-server ansible]# ansible-playbook metrics_role.yml
16.2 验证metrics
[[root@k8s-master01 ~]# kubectl get pod -n kube-system |grep metrics
metrics-server-556774f65b-5wk4m 1/1 Running 0 31s[root@k8s-master01 ~]# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master01.example.local 155m 7% 1715Mi 47%
k8s-master02.example.local 105m 5% 1320Mi 36%
k8s-master03.example.local 120m 6% 1428Mi 39%
k8s-node01.example.local 45m 2% 935Mi 25%
k8s-node02.example.local 62m 3% 983Mi 27%
k8s-node03.example.local 48m 2% 922Mi 25%
17.安装dashboard
17.1 安装dashboard
[root@ansible-server ansible]# mkdir -p roles/dashboard/{tasks,vars,files,templates}
[root@ansible-server ansible]# cd roles/dashboard/
[root@ansible-server dashboard]# ls
files tasks templates vars#下面HARBOR_DOMAIN的地址设置成自己的harbor域名地址
[root@ansible-server dashboard]# vim vars/main.yml
HARBOR_DOMAIN: harbor.raymonds.cc
USERNAME: admin
PASSWORD: 123456
NODEPORT: 30005[root@ansible-server dashboard]# cat templates/recommended.yaml.j2
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.apiVersion: v1
kind: Namespace
metadata:name: kubernetes-dashboard---apiVersion: v1
kind: ServiceAccount
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard---kind: Service
apiVersion: v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard
spec:ports:- port: 443targetPort: 8443selector:k8s-app: kubernetes-dashboardtype: NodePort---apiVersion: v1
kind: Secret
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboard-certsnamespace: kubernetes-dashboard
type: Opaque---apiVersion: v1
kind: Secret
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboard-csrfnamespace: kubernetes-dashboard
type: Opaque
data:csrf: ""---apiVersion: v1
kind: Secret
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboard-key-holdernamespace: kubernetes-dashboard
type: Opaque---kind: ConfigMap
apiVersion: v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboard-settingsnamespace: kubernetes-dashboard---kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard
rules:# Allow Dashboard to get, update and delete Dashboard exclusive secrets.- apiGroups: [""]resources: ["secrets"]resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]verbs: ["get", "update", "delete"]# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.- apiGroups: [""]resources: ["configmaps"]resourceNames: ["kubernetes-dashboard-settings"]verbs: ["get", "update"]# Allow Dashboard to get metrics.- apiGroups: [""]resources: ["services"]resourceNames: ["heapster", "dashboard-metrics-scraper"]verbs: ["proxy"]- apiGroups: [""]resources: ["services/proxy"]resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]verbs: ["get"]---kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboard
rules:# Allow Metrics Scraper to get metrics from the Metrics server- apiGroups: ["metrics.k8s.io"]resources: ["pods", "nodes"]verbs: ["get", "list", "watch"]---apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: kubernetes-dashboard
subjects:- kind: ServiceAccountname: kubernetes-dashboardnamespace: kubernetes-dashboard---apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: kubernetes-dashboard
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: kubernetes-dashboard
subjects:- kind: ServiceAccountname: kubernetes-dashboardnamespace: kubernetes-dashboard---kind: Deployment
apiVersion: apps/v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard
spec:replicas: 1revisionHistoryLimit: 10selector:matchLabels:k8s-app: kubernetes-dashboardtemplate:metadata:labels:k8s-app: kubernetes-dashboardspec:containers:- name: kubernetes-dashboardimage: kubernetesui/dashboard:v2.4.0imagePullPolicy: Alwaysports:- containerPort: 8443protocol: TCPargs:- --auto-generate-certificates- --namespace=kubernetes-dashboard# Uncomment the following line to manually specify Kubernetes API server Host# If not specified, Dashboard will attempt to auto discover the API server and connect# to it. Uncomment only if the default does not work.# - --apiserver-host=http://my-address:portvolumeMounts:- name: kubernetes-dashboard-certsmountPath: /certs# Create on-disk volume to store exec logs- mountPath: /tmpname: tmp-volumelivenessProbe:httpGet:scheme: HTTPSpath: /port: 8443initialDelaySeconds: 30timeoutSeconds: 30securityContext:allowPrivilegeEscalation: falsereadOnlyRootFilesystem: truerunAsUser: 1001runAsGroup: 2001volumes:- name: kubernetes-dashboard-certssecret:secretName: kubernetes-dashboard-certs- name: tmp-volumeemptyDir: {}serviceAccountName: kubernetes-dashboardnodeSelector:"kubernetes.io/os": linux# Comment the following tolerations if Dashboard must not be deployed on mastertolerations:- key: node-role.kubernetes.io/mastereffect: NoSchedule---kind: Service
apiVersion: v1
metadata:labels:k8s-app: dashboard-metrics-scrapername: dashboard-metrics-scrapernamespace: kubernetes-dashboard
spec:ports:- port: 8000targetPort: 8000selector:k8s-app: dashboard-metrics-scraper---kind: Deployment
apiVersion: apps/v1
metadata:labels:k8s-app: dashboard-metrics-scrapername: dashboard-metrics-scrapernamespace: kubernetes-dashboard
spec:replicas: 1revisionHistoryLimit: 10selector:matchLabels:k8s-app: dashboard-metrics-scrapertemplate:metadata:labels:k8s-app: dashboard-metrics-scraperspec:securityContext:seccompProfile:type: RuntimeDefaultcontainers:- name: dashboard-metrics-scraperimage: kubernetesui/metrics-scraper:v1.0.7ports:- containerPort: 8000protocol: TCPlivenessProbe:httpGet:scheme: HTTPpath: /port: 8000initialDelaySeconds: 30timeoutSeconds: 30volumeMounts:- mountPath: /tmpname: tmp-volumesecurityContext:allowPrivilegeEscalation: falsereadOnlyRootFilesystem: truerunAsUser: 1001runAsGroup: 2001serviceAccountName: kubernetes-dashboardnodeSelector:"kubernetes.io/os": linux# Comment the following tolerations if Dashboard must not be deployed on mastertolerations:- key: node-role.kubernetes.io/mastereffect: NoSchedulevolumes:- name: tmp-volumeemptyDir: {}[root@ansible-server dashboard]# vim templates/recommended.yaml.j2
...
kind: Service
apiVersion: v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard
spec:type: NodePort #添加这行ports:- port: 443targetPort: 8443nodePort: {{ NODEPORT }} #添加这行selector:k8s-app: kubernetes-dashboard
...[root@ansible-server dashboard]# vim files/admin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:name: admin-usernamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: name: admin-userannotations:rbac.authorization.kubernetes.io/autoupdate: "true"
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: cluster-admin
subjects:
- kind: ServiceAccountname: admin-usernamespace: kube-system[root@ansible-server dashboard]# vim tasks/dashboard_file.yml
- name: copy recommended.yaml filetemplate:src: recommended.yaml.j2dest: /root/recommended.yaml
- name: copy admin.yaml filecopy:src: admin.yamldest: /root/admin.yaml[root@ansible-server dashboard]# vim tasks/config.yml
- name: Modify the "image:" linereplace:path: /root/recommended.yamlregexp: '(.*image:) kubernetesui(/.*)'replace: '\1 {{ HARBOR_DOMAIN }}/google_containers\2'[root@ansible-server dashboard]# vim tasks/download_images.yml
- name: get dashboard versionshell:chdir: /rootcmd: awk -F "/" '/image:/{print $NF}' recommended.yamlregister: DASHBOARD_VERSION
- name: download dashboard imageshell:cmd: |{% for i in DASHBOARD_VERSION.stdout_lines %}ctr images pull --all-platforms registry.aliyuncs.com/google_containers/{{ i }}ctr images tag registry.aliyuncs.com/google_containers/{{ i }} {{ HARBOR_DOMAIN }}/google_containers/{{ i }}ctr images remove registry.aliyuncs.com/google_containers/{{ i }}expect <<EOFspawn ctr images push --plain-http -u {{ USERNAME }} {{ HARBOR_DOMAIN }}/google_containers/{{ i }}expect "Password:" { send "{{ PASSWORD }}\n";exp_continue }EOF{% endfor %}[root@ansible-server dashboard]# vim tasks/install_dashboard.yml
- name: install dashboardshell:chdir: /rootcmd: "kubectl --kubeconfig=/etc/kubernetes/admin.conf apply -f recommended.yaml -f admin.yaml"[root@ansible-server dashboard]# vim tasks/main.yml
- include: dashboard_file.yml
- include: config.yml
- include: download_images.yml
- include: install_dashboard.yml[root@ansible-server dashboard]# cd ../../
[root@ansible-server ansible]# tree roles/dashboard/
roles/dashboard/
├── files
│ └── admin.yaml
├── tasks
│ ├── config.yml
│ ├── dashboard_file.yml
│ ├── download_images.yml
│ ├── install_dashboard.yml
│ └── main.yml
├── templates
│ └── recommended.yaml.j2
└── vars└── main.yml4 directories, 8 files[root@ansible-server ansible]# vim dashboard_role.yml
---
- hosts: master01roles:- role: dashboard[root@ansible-server ansible]# ansible-playbook dashboard_role.yml
17.2 登录dashboard
https://172.31.3.101:30005
查看token值：
[root@k8s-master01 ~]# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
Name: admin-user-token-8l79p
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name: admin-userkubernetes.io/service-account.uid: 4b32f344-9ede-4093-8e8c-f9df665177caType: kubernetes.io/service-account-tokenData
====
ca.crt: 1099 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IkNmN1ZHUU5Vd2d5bV9hQnUtcWtjN005Q1NGb29oaXgzWTR5OVJWRVFydmsifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLThsNzlwIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI0YjMyZjM0NC05ZWRlLTQwOTMtOGU4Yy1mOWRmNjY1MTc3Y2EiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.rYVh5KxoWOLtE0r6VyNMTjb7GG8ATWqOWsRjwVbpUJyWOFteViKn91buEjXGp2_NxTsafb9Ck2C6f4W-Q3irI16zJp8lh18FPHZRw6XrpIKv8gaB0MKc_BmmYuIgGA-mFRracL1ZaW9uwa_DM_s0s4pUev2vl5Ii7cSGjihrCnD5aXejP1tQK_XwZ8PqrXAToUBB9Yy9oRWS6qhSsRDWmKMDvOMUGngmPyLQM7_0mDMQRI2LSRRFqVHup7cWYqUoaZOuIlz0NB8JyipSYQItB6wjJy6SwXCOCyvtZ8sasEbP6_UoixvfIbD2Gqv148Xfnol6RUceodah2cHne4O90A
a36.ansible 生产实战案例 -- 基于kubeadm安装kubernetes v1.23 -- 集群部署(一)相关推荐
- a32.ansible 生产实战案例 -- 基于kubeadm安装kubernetes v1.22 -- 集群部署(一)
源码下载地址:https://github.com/raymond999999/kubernetes-ansible 1.高可用Kubernetes集群规划 角色 机器名 机器配置 ip地址 安装软件 ...
- a24.ansible 生产实战案例 -- 基于kubeadm安装kubernetes v1.20 -- 集群部署(一)
源码下载地址:https://github.com/raymond999999/kubernetes-ansible 1.高可用Kubernetes集群规划 角色 机器名 机器配置 ip地址 安装软件 ...
- kubeadm安装Kubernetes V1.10集群详细文档
1:服务器信息以及节点介绍 系统信息:centos1708 minimal 只修改IP地址 主机名称 IP 备注 node01 192.168.150.181 master and etcd r ...
- 使用kubeadm安装kubernetes高可用集群
kubeadm安装kubernetes高可用集群搭建 第一步:首先搭建etcd集群 yum install -y etcd 配置文件 /etc/etcd/etcd.confETCD_NAME=inf ...
- 使用 kubeadm 创建 kubernetes 1.9 集群
简介 kubeadm是一个kubernetes官方提供的快速安装和初始化拥有最佳实践(best practice)的kubernetes集群的工具,虽然目前还处于 beta 和 alpha 状态,还不 ...
- Centos7.4 安装elasticsearch6.1.3集群部署
Centos7.4 安装elasticsearch6.1.3集群部署 安装elasticsearch 1.依赖环境安装 这里使用的java 是1.8.0_77的版本.使用的是rpm 安装包的形式进行部 ...
- p2p 文件服务器集群,基于云计算的P2P流媒体服务器集群部署算法.doc
基于云计算的P2P流媒体服务器集群部署算法.doc 基于云计算的P2P流媒体服务器集群部署算法 摘 要: 针对云计算数据中心网络(DCN)环境下,P2P流媒体服务器集群部署引起的较高带宽占用问题,提出 ...
- 深入玩转K8S之使用kubeadm安装Kubernetes v1.10以及常见问题解答
原文链接:http://blog.51cto.com/devingeng/2096495 关于K8S: Kubernetes是Google开源的容器集群管理系统.它构建于docker技术之上,为容器化 ...
- 二、《云原生 | Kubernetes篇》Kubernetes学习-K8S安装篇-Kubeadm安装高可用K8S集群--生产环境
目录 1. Kubernetes 高可用安装 1.1.1实验环境规划 高可用Kubernetes集群规划
- Kubeadm安装k8s高可用集群实战
请不要使用带中文的服务器和克隆的虚拟机! 文档中的IP地址请统一替换,不要一个一个替换! 一.集群安装网段划分 集群安装时会涉及到三个网段: 宿主机网段:就是安装k8s的服务器 Pod网段:k8s P ...
最新文章
- Operation Queues并发编程
- 《阿丽塔》脑机接口正从科幻走向现实
- CF908D New Year and Arbitrary Arrangement
- 红黑树效率为甚恶魔是log_一文带你彻底读懂红黑树（附详细图解）
- select fd_set
- Github标星3.7k：微软出的数据科学入门课程（附下载）
- boost::mpl模块实现pair_view相关的测试程序
- NBU网络备份大全之oracle系统冷备份
- 在博客园的博客插入代码块（高亮、行号、缩进、着色）
- web文件管理系统_财务影像管理系统
- SSM : 环境搭建
- 二项分布的实现（np.random.binomial）
- SQL语法--DML
- RK30SDK开发板驱动分析（一）：platform device 的概念与注册
- BBS论坛系统需求说明书
- 社交网络图形可视化工具Gephi使用教程
- eclipse加载jdbc驱动(mysql)
- 有哪些写项目策划书注意事项
- 什么是悬空 Docker 镜像？
- [每日一氵] linux 批量删除某个名字的文件夹