二进制安装Kubernetes（k8s） v1.24.0 IPv4
2024-06-01 10:56:33
感谢：二进制安装Kubernetes（k8s） v1.24.0 IPv4/IPv6双栈 - 小陈运维
kubernetes 1.24 变化较大，详细见：Kubernetes 1.24 的删除和弃用 | Kubernetes
1.k8s基础系统环境配置
宿主机、部署软件：
为了节省资源，将Lb01中的资源部署到Node01、将Lb02中的资源部署到Node02，建议k8s集群与etcd集群分开安装
| 主机名称 | IP地址 | 说明 | 软件 |
|---|---|---|---|
| Master01 | 10.4.7.11 | master节点 |
kube-apiserver、kube-controller-manager、kube-scheduler、etcd、 kubelet、kube-proxy、nfs-client |
| Master02 | 10.4.7.12 | master节点 |
kube-apiserver、kube-controller-manager、kube-scheduler、etcd、 kubelet、kube-proxy、nfs-client |
| Master03 | 10.4.7.21 | master节点 |
kube-apiserver、kube-controller-manager、kube-scheduler、etcd、 kubelet、kube-proxy、nfs-client |
| Node01 | 10.4.7.22 | node节点 | kubelet、kube-proxy、nfs-client、haproxy、keepalived |
| Node02 | 10.4.7.200 | node节点 | kubelet、kube-proxy、nfs-client、haproxy、keepalived |
| 10.4.7.10 | VIP |
宿主机要求：
宿主机要求：必须是Centos8以上，因为 k8s v1.24.0 在CentOS 7有兼容性问题，会导致有些字段不识别kubelet Error getting node 问题求助_Jerry00713的博客-CSDN博客
| 软件 | 版本 |
|---|---|
| 内核 | 5.17.5-1.el8.elrepo 以上 |
| CentOS 8 | v8 或者 v7 |
| kube-apiserver、kube-controller-manager、kube-scheduler、kubelet、kube-proxy | v1.24.0 |
| etcd | v3.5.4 |
| containerd | v1.5.11 |
| cfssl | v1.6.1 |
| cni | v1.1.1 |
| crictl | v1.23.0 |
| haproxy | v1.8.27 |
| keepalived | v2.1.5 |
网段：
宿主机：10.4.7.0/24
k8s 集群IP(clusterIP)：192.168.0.0/16
k8s pod集群(cluster-cidr)：172.7.0.0/16
别名：
(all) 代表 k8s-master01、k8s-master02、k8s-master03、k8s-node01、k8s--node2
(k8s_all) 代表 k8s-master01、k8s-master02、k8s-master03、k8s-node01、k8s--node2
(k8s_master) 代表 k8s-master01、k8s-master02、k8s-master03
(k8s_node) 代表 k8s-node01、k8s--node2
(Lb) 正常找另外两台设备，作为负载均衡、VIP等操作，此文章为节约资源，部署到 k8s-node01、k8s--node2 中
1.1.配置IP (all)
略自己配置宿主机IP
1.2.设置主机名 (all)
10.4.7.11 hostnamectl set-hostname k8s-master01
10.4.7.12 hostnamectl set-hostname k8s-master02
10.4.7.21 hostnamectl set-hostname k8s-master03
10.4.7.22 hostnamectl set-hostname k8s-node01
10.4.7.200 hostnamectl set-hostname k8s-node02
1.3.配置yum源 (all)
1.修改 vi /etc/yum.repos.d/CentOS-Linux-BaseOS.repo 文件
将mirrorlist配置注释掉，并将baseurl修改为阿里云镜像地址
[BaseOS]
name=CentOS-$releasever - Base
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=BaseOS&infra=$infra
baseurl=https://mirrors.aliyun.com/centos/$releasever-stream/BaseOS/$basearch/os/
http://mirrors.aliyuncs.com/centos/$releasever-stream/BaseOS/$basearch/os/
http://mirrors.cloud.aliyuncs.com/centos/$releasever-stream/BaseOS/$basearch/os/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
2.修改 vi /etc/yum.repos.d/CentOS-Linux-AppStream.repo 文件
将mirrorlist配置注释掉，并将baseurl修改为阿里云镜像地址
[AppStream]
name=CentOS-$releasever - AppStream
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=AppStream&infra=$infra
baseurl=https://mirrors.aliyun.com/centos/$releasever-stream/AppStream/$basearch/os/
http://mirrors.aliyuncs.com/centos/$releasever-stream/AppStream/$basearch/os/
http://mirrors.cloud.aliyuncs.com/centos/$releasever-stream/AppStream/$basearch/os/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
3.修改 vi /etc/yum.repos.d/CentOS-Linux-Extras.repo 文件
将mirrorlist配置注释掉，并将baseurl修改为阿里云镜像地址[extras]
name=CentOS-$releasever - Extras
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
baseurl=https://mirrors.aliyun.com/centos/$releasever-stream/extras/$basearch/os/
http://mirrors.aliyuncs.com/centos/$releasever-stream/extras/$basearch/os/
http://mirrors.cloud.aliyuncs.com/centos/$releasever-stream/extras/$basearch/os/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
yum clean all && yum makecache
写成脚本：
sed -e 's|^mirrorlist=|#mirrorlist=|g' -e '/#baseurl=/d' -e '/#mirrorlist/a\baseurl=https://mirrors.aliyun.com/centos/$releasever-stream/BaseOS/$basearch/os/\n http://mirrors.aliyuncs.com/centos/$releasever-stream/BaseOS/$basearch/os/\n http://mirrors.cloud.aliyuncs.com/centos/$releasever-stream/BaseOS/$basearch/os/' -i /etc/yum.repos.d/CentOS-Linux-BaseOS.reposed -e 's|^mirrorlist=|#mirrorlist=|g' -e '/#baseurl=/d' -e '/#mirrorlist/a\baseurl=https://mirrors.aliyun.com/centos/$releasever-stream/AppStream/$basearch/os/\n http://mirrors.aliyuncs.com/centos/$releasever-stream/AppStream/$basearch/os/\n http://mirrors.cloud.aliyuncs.com/centos/$releasever-stream/AppStream/$basearch/os/' -i /etc/yum.repos.d/CentOS-Linux-AppStream.reposed -e 's|^mirrorlist=|#mirrorlist=|g' -e '/#baseurl=/d' -e '/#mirrorlist/a\baseurl=https://mirrors.aliyun.com/centos/$releasever-stream/extras/$basearch/os/\n http://mirrors.aliyuncs.com/centos/$releasever-stream/extras/$basearch/os/\n http://mirrors.cloud.aliyuncs.com/centos/$releasever-stream/extras/$basearch/os/' -i /etc/yum.repos.d/CentOS-Linux-Extras.repoyum clean all && yum makecache1.4.安装一些必备工具 (all)
yum -y install vim wget jq psmisc vim net-tools nfs-utils telnet yum-utils device-mapper-persistent-data lvm2 git network-scripts tar curl -y1.5.下载需要工具
mkdir /opt/src/;cd /opt/src/
1.下载kubernetes1.24.+的二进制包 (在master01，通过master01发送给各个k8s节点)
github二进制包下载地址：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.mdwget https://dl.k8s.io/v1.24.0/kubernetes-server-linux-amd64.tar.gz
2.下载etcdctl二进制包 (etcd节点，此文章etcd部署在k8s-master01、k8s-master02、k8s-master03)
github二进制包下载地址：https://github.com/etcd-io/etcd/releaseswget https://github.com/etcd-io/etcd/releases/download/v3.5.4/etcd-v3.5.4-linux-amd64.tar.gz
3.docker-ce二进制包下载地址 (k8s_all)
二进制包下载地址：https://download.docker.com/linux/static/stable/x86_64/这里需要下载20.10.+版本
wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.14.tgz
4.containerd二进制包下载 (k8s_all)
github下载地址：https://github.com/containerd/containerd/releasescontainerd下载时下载带cni插件的二进制包。
wget https://github.com/containerd/containerd/releases/download/v1.6.4/cri-containerd-cni-1.6.4-linux-amd64.tar.gz
5.下载cfssl二进制包 (master01，通过master01制作证书发送给各个k8s节点)
github二进制包下载地址：https://github.com/cloudflare/cfssl/releaseswget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd646.cni插件下载 (k8s_all)
github下载地址：https://github.com/containernetworking/plugins/releaseswget https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz
7.crictl客户端二进制下载 (k8s_all)
github下载：https://github.com/kubernetes-sigs/cri-tools/releaseswget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.23.0/crictl-v1.23.0-linux-amd64.tar.gz
1.6.关闭防火墙、SELinux (all)
systemctl disable --now firewalld
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config关闭防火墙、selinux写成一句话：
systemctl stop firewalld.service;systemctl disable firewalld.service;sed -i -r 's#(^SELIN.*=)enforcing#\1disable#g' /etc/selinux/config;setenforce 0
1.7.关闭交换分区 (all)
sed -ri 's/.*swap.*/#&/' /etc/fstab
swapoff -a && sysctl -w vm.swappiness=0cat /etc/fstab
# /dev/mapper/centos-swap swap swap defaults 0 01.8.关闭NetworkManager 并启用 network (k8s_all、Lb可不做)
systemctl disable --now NetworkManager
systemctl start network && systemctl enable network1.9.进行时间同步 (Lb可不做)
# 找一台机器做服务端、或者选择matser01也可，让作为服务端pool更新阿里云时间
yum install chrony -y
cat > /etc/chrony.conf << EOF
pool ntp.aliyun.com iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
allow 10.4.7.0/24
local stratum 10
keyfile /etc/chrony.keys
leapsectz right/UTC
logdir /var/log/chrony
EOFsystemctl restart chronyd
systemctl enable chronyd# 客户端，要保证所k8s-master、k8s-node的时间都一致
yum install chrony -y
vim /etc/chrony.conf
cat /etc/chrony.conf | grep -v "^#" | grep -v "^$"
pool 10.4.7.11 iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
keyfile /etc/chrony.keys
leapsectz right/UTC
logdir /var/log/chronysystemctl restart chronyd ; systemctl enable chronyd# 客户端安装一条命令
yum install chrony -y ; sed -i "s#2.centos.pool.ntp.org#10.4.7.11#g" /etc/chrony.conf ; systemctl restart chronyd ; systemctl enable chronyd#使用客户端进行验证
chronyc sources -v1.10.配置ulimit (k8s_all、Lb可不做)
ulimit -SHn 655350
cat >> /etc/security/limits.conf <<EOF
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* seft memlock unlimited
* hard memlock unlimitedd
EOF1.11.配置免密登录 (master01节点)
在master01节点做ssh密钥对，拷贝ssh_pub到所有节点，其中SSHPASS为各个节点的统一密码，StrictHostKeyChecking=no取消交互式询问（也就是yes\no）的提示，sshpass进行自动登录分发密钥
yum install -y sshpass
ssh-keygen -f /root/.ssh/id_rsa -P ''
export IP="10.4.7.12 10.4.7.21 10.4.7.22 10.4.7.200"
export SSHPASS=123456
for HOST in $IP;do sshpass -e ssh-copy-id -o StrictHostKeyChecking=no $HOST;done1.12.添加启用源 (Lb可不做)
# 为 RHEL-8或 CentOS-8配置源
yum install -y https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm# 查看可用安装包
yum --disablerepo="*" --enablerepo="elrepo-kernel" list available1.13.升级内核至4.18版本以上 (Lb可不做)
# 查看当前使用的内核
[root@localhost ~]# grubby --default-kernel
/boot/vmlinuz-4.18.0-348.el8.x86_64# 查看已安装那些内核
[root@localhost ~]# rpm -qa | grep kernel
kernel-modules-4.18.0-348.el8.x86_64
kernel-tools-4.18.0-348.el8.x86_64
kernel-4.18.0-348.el8.x86_64
kernel-core-4.18.0-348.el8.x86_64
kernel-tools-libs-4.18.0-348.el8.x86_64
[root@localhost yum.repos.d]# # 安装最新的内核
# 我这里选择的是稳定版kernel-ml 如需更新长期维护版本kernel-lt
[root@localhost ~]# yum --enablerepo=elrepo-kernel install kernel-ml -y# 查看已安装那些内核
[root@localhost ~]# rpm -qa | grep kernel
kernel-modules-4.18.0-348.el8.x86_64
kernel-tools-4.18.0-348.el8.x86_64
kernel-4.18.0-348.el8.x86_64
kernel-ml-core-5.18.3-1.el8.elrepo.x86_64
kernel-ml-5.18.3-1.el8.elrepo.x86_64 （此内核为刚刚yum安装的）
kernel-core-4.18.0-348.el8.x86_64
kernel-tools-libs-4.18.0-348.el8.x86_64
kernel-ml-modules-5.18.3-1.el8.elrepo.x86_64# 查看默认内核
grubby --default-kernel
/boot/vmlinuz-5.16.7-1.el8.elrepo.x86_64# 若不是最新的使用命令设置
grubby --set-default /boot/vmlinuz-「您的内核版本」.x86_64
grubby --set-default "/boot/vmlinuz-5.18.3-1.el8.elrepo.x86_64"# 重启生效
reboot
1.14.安装ipvsadm (Lb可不做)
yum install ipvsadm ipset sysstat conntrack libseccomp -ycat >> /etc/modules-load.d/ipvs.conf <<EOF
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOFsystemctl restart systemd-modules-load.servicelsmod | grep -e ip_vs -e nf_conntrack
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs 180224 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack 176128 1 ip_vs
nf_defrag_ipv6 24576 2 nf_conntrack,ip_vs
nf_defrag_ipv4 16384 1 nf_conntrack
libcrc32c 16384 3 nf_conntrack,xfs,ip_vs1.15.修改内核参数 (Lb可不做)
cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.all.forwarding = 1EOFsysctl --system1.16.所有节点搭建配置域名解析 (all)
域名解析的作用：
1、让各个pc机器能够互相通过域名访问彼此
2、外网访问一个网址，将流量通过域名解析抛给到apiserver，在通过ingress控制器转发到pod中，实现外网能够通过访问网址进行访问pod
[root@k8s-master01 ~]# yum install bind bind9.16-utils -y
[root@k8s-master01 ~]# vi /etc/named.conf # 修改options下的内容，注释去掉options {listen-on port 53 { 10.4.7.11; }; // 默认监听53端口，IP改成自己的本机的IP，要让内网的所有机器能够访问我的bind进行dns解析，127.0.0.1只能自己用，注意bind语法严格，所以格式要写对listen-on-v6 port 53 { ::1; }; //去掉这行，不需要IPV6directory "/var/named";dump-file "/var/named/data/cache_dump.db";statistics-file "/var/named/data/named_stats.txt";memstatistics-file "/var/named/data/named_mem_stats.txt";recursing-file "/var/named/data/named.recursing";secroots-file "/var/named/data/named.secroots";allow-query { any; }; // 改成any，让任何人都可以访问你forwarders { 10.4.7.254; }; // 增加一条，如果dns解析不了，王上一级dns，也就是网关 recursion yes; //递归模式dnssec-enable no; //是否支持DNSSEC开关,默认为yes。 dnssec-validation no; //是否进行DNSSEC确认开关,默认为no。dnssec-enable配置项用来设置是否启用DNSSEC支持，DNS安全扩展（DNSSEC）提供了验证DNS数据有效性的系统。[root@k8s-master01 ~]# named-checkconf # 检查是否有问题。无任何提示代表无问题
[root@k8s-master01 ~]# vi /etc/named.rfc1912.zones # 配置区域文件，在最末尾增加两条，注释去掉
# 主机域，在内部自己使用，比如物理机(10.4.7.11 ping 10.4.7.11） # 可以使用ping hdss7-12.host.com，所以host是可以变化的 zone "host.com" IN {type master;file "host.com.zone";allow-update { 10.4.7.11; }; };# 业务域，把流量引入到k8s中，比如访问jenkins，页面地址输入jenkins.od.com # 所以od是可以变化的，按照公司要求 zone "od.com" IN {type master;file "od.com.zone";allow-update { 10.4.7.11; }; };
编写主机域的配置文件，需要域名解析的主机都写入才可通过域名访问：
[root@k8s-master01 ~]# vi /var/named/host.com.zone$ORIGIN host.com. $TTL 600 ; 10 minutes @ IN SOA dns.host.com. dnsadmin.host.com. (2020010501 ; serial10800 ; refresh (3 hours)900 ; retry (15 minutes)604800 ; expire (1 week)86400 ; minimum (1 day))NS dns.host.com. $TTL 60 ; 1 minute dns A 10.4.7.11 k8s-master01 A 10.4.7.11 k8s-master02 A 10.4.7.12 k8s-master03 A 10.4.7.21 k8s-node01 A 10.4.7.22 k8s-node02 A 10.4.7.200 lb-vip A 10.4.7.10
业务域配置文件，流量经过bind，通过业务域配置文件中对应的关系，解析出对应的IP：
[root@k8s-master01 ~]# vi /var/named/od.com.zone$ORIGIN od.com. $TTL 600 ; 10 minutes @ IN SOA dns.od.com. dnsadmin.od.com. (2020010501 ; serial10800 ; refresh (3 hours)900 ; retry (15 minutes)604800 ; expire (1 week)86400 ; minimum (1 day))NS dns.od.com. $TTL 60 ; 1 minute dns A 10.4.7.11[root@k8s-master01 ~]# named-checkconf 检查是否有问题。无任何提示代表无问题
启动named服务:
[root@k8s-master01 ~]# systemctl start named
[root@k8s-master01 ~]# systemctl enable named
测试:
[root@k8s-master01 ~]# dig -t A k8s-master02.host.com @10.4.7.11 +short # 代表主机域无问题
10.4.7.12
对所有的机器的网卡，配置DNS为10.4.7.11：
涉及机器：（hdss7-11、hdss7-12、hdss7-21、hdss7-22、hdss7-200）
vi /etc/sysconfig/network-scripts/ifcfg-ens33 |grep "DNS1"
DNS1=10.4.7.11
重启网卡：systemctl restart network
测试：ping k8s-master02.host.com 如果可以说明已经生效
[root@k8s-master01 ~]# ping k8s-master02.host.com
PING k8s-master02.host.com (10.4.7.12) 56(84) bytes of data.
64 bytes from 10.4.7.12 (10.4.7.12): icmp_seq=1 ttl=64 time=0.503 ms
64 bytes from 10.4.7.12 (10.4.7.12): icmp_seq=2 ttl=64 time=0.709 ms
实现短域名（不带host.com）
[root@k8s-master01 ~]# ping k8s-master02
ping: k8s-master02: 未知的名称或服务
涉及机器：（hdss7-11、hdss7-12、hdss7-21、hdss7-22、hdss7-200）
vi /etc/resolv.conf # 添加search host.com，比如hdss7-11的esolv.conf
# Generated by NetworkManager
search host.com
nameserver 10.4.7.11测试：ping hdss7-12 等等
[root@k8s-master01 ~]# ping k8s-master02
PING k8s-master02.host.com (10.4.7.12) 56(84) bytes of data.
64 bytes from 10.4.7.12 (10.4.7.12): icmp_seq=1 ttl=64 time=0.666 ms
64 bytes from 10.4.7.12 (10.4.7.12): icmp_seq=2 ttl=64 time=0.801 ms
问题：重启后，resolv.conf会恢复源文件，导致search host.com丢失
涉及机器：（hdss7-11、hdss7-12、hdss7-21、hdss7-22、hdss7-200）
[root@k8s-master01 ~]# vi /etc/rc.d/rc.local # 开机向resolv.conf 插入短域名，注意脚本提权
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# 请注意，您必须运行'chmod+x/etc/rc.d/rc.local“以确保
sed -i "2isearch host.com" /etc/resolv.conf [root@k8s-master01 ~]# chmod +x /etc/rc.d/rc.local
写成一句话：sed -i '$a\sed -i "2isearch host.com" /etc/resolv.conf' /etc/rc.d/rc.local;chmod +x /etc/rc.d/rc.local
2.k8s基本组件安装
2.1.所有k8s节点安装Containerd作为Runtime (k8s_all)
cd /opt/src#创建cni插件所需目录
mkdir -p /etc/cni/net.d /opt/cni/bin
#解压cni二进制包
tar xf cni-plugins-linux-amd64-v1.1.1.tgz -C /opt/cni/bin/#解压
tar -C / -xzf cri-containerd-cni-1.6.4-linux-amd64.tar.gz#创建服务启动文件
cat > /etc/systemd/system/containerd.service <<EOF
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
TasksMax=infinity
OOMScoreAdjust=-999[Install]
WantedBy=multi-user.target
EOF2.1.1配置Containerd所需的模块
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF2.1.2加载模块
systemctl restart systemd-modules-load.service2.1.3配置Containerd所需的内核
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF# 加载内核sysctl --system2.1.4创建Containerd的配置文件
mkdir -p /etc/containerd
containerd config default | tee /etc/containerd/config.toml修改Containerd的配置文件
sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml# 找到containerd.runtimes.runc.options，要存在SystemdCgroup = true
[root@k8s-master01 src]# grep -B 13 -A 2 "SystemdCgroup" /etc/containerd/config.toml runtime_type = "io.containerd.runc.v2"[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]BinaryName = ""CriuImagePath = ""CriuPath = ""CriuWorkPath = ""IoGid = 0IoUid = 0NoNewKeyring = falseNoPivotRoot = falseRoot = ""ShimCgroup = ""SystemdCgroup = true[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
[root@k8s-master01 src]# # 将/etc/containerd/config.toml 的sandbox_image默认地址改为符合版本地址
[root@k8s-master01 src]# grep -C 2 "sandbox_image" /etc/containerd/config.tomlnetns_mounts_under_state_dir = falserestrict_oom_score_adj = false
# sandbox_image = "k8s.gcr.io/pause:3.6"sandbox_image = "registry.cn-hangzhou.aliyuncs.com/chenby/pause:3.6"selinux_category_range = 1024stats_collect_period = 102.1.5启动并设置为开机启动
systemctl daemon-reload
systemctl enable --now containerd2.1.6配置crictl客户端连接的运行时位置
#解压
tar xf crictl-v1.23.0-linux-amd64.tar.gz -C /usr/bin/
#生成配置文件cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF#测试
systemctl restart containerd
crictl info
2.2.k8s与etcd下载及安装（仅在master01操作）
2.2.1解压k8s安装包
# 解压k8s安装文件
cd /opt/src
tar -xf kubernetes-server-linux-amd64.tar.gz --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}# 解压etcd安装文件
tar -xf etcd-v3.5.4-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.5.4-linux-amd64/etcd{,ctl}# 查看/usr/local/bin下内容
[root@k8s-master01 src]# ls /usr/local/bin/
containerd containerd-shim-runc-v1 containerd-stress critest ctr etcdctl kube-controller-manager kubelet kube-scheduler
containerd-shim containerd-shim-runc-v2 crictl ctd-decoder etcd kube-apiserver kubectl kube-proxy2.2.2查看版本
[root@k8s-master01 src]# kubelet --version
Kubernetes v1.24.0
[root@k8s-master01 src]# etcdctl version
etcdctl version: 3.5.4
API version: 3.5
[root@k8s-master01 src]# 2.2.3将组件发送至其他k8s节点
Master='k8s-master02 k8s-master03'
Work='k8s-node01 k8s-node02'for NODE in $Master; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; donefor NODE in $Work; do scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done
2.3创建证书相关文件 （仅在master01操作）
mkdir /root/pki;cd /root/pki
cat > admin-csr.json << EOF
{"CN": "admin","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "system:masters","OU": "Kubernetes-manual"}]
}
EOFcat > ca-config.json << EOF
{"signing": {"default": {"expiry": "876000h"},"profiles": {"kubernetes": {"usages": ["signing","key encipherment","server auth","client auth"],"expiry": "876000h"}}}
}
EOFcat > etcd-ca-csr.json << EOF
{"CN": "etcd","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "etcd","OU": "Etcd Security"}],"ca": {"expiry": "876000h"}
}
EOFcat > front-proxy-ca-csr.json << EOF
{"CN": "kubernetes","key": {"algo": "rsa","size": 2048},"ca": {"expiry": "876000h"}
}
EOFcat > kubelet-csr.json << EOF
{"CN": "system:node:$NODE","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "Beijing","ST": "Beijing","O": "system:nodes","OU": "Kubernetes-manual"}]
}
EOFcat > manager-csr.json << EOF
{"CN": "system:kube-controller-manager","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "system:kube-controller-manager","OU": "Kubernetes-manual"}]
}
EOFcat > apiserver-csr.json << EOF
{"CN": "kube-apiserver","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "Kubernetes","OU": "Kubernetes-manual"}]
}
EOFcat > ca-csr.json << EOF
{"CN": "kubernetes","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "Kubernetes","OU": "Kubernetes-manual"}],"ca": {"expiry": "876000h"}
}
EOFcat > etcd-csr.json << EOF
{"CN": "etcd","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "etcd","OU": "Etcd Security"}]
}
EOFcat > front-proxy-client-csr.json << EOF
{"CN": "front-proxy-client","key": {"algo": "rsa","size": 2048}
}
EOFcat > kube-proxy-csr.json << EOF
{"CN": "system:kube-proxy","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "system:kube-proxy","OU": "Kubernetes-manual"}]
}
EOFcat > scheduler-csr.json << EOF
{"CN": "system:kube-scheduler","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "system:kube-scheduler","OU": "Kubernetes-manual"}]
}
EOFmkdir /root/bootstrap;cd /root/bootstrap
cat > bootstrap.secret.yaml << EOF
apiVersion: v1
kind: Secret
metadata:name: bootstrap-token-c8ad9cnamespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:description: "The default bootstrap token generated by 'kubelet '."token-id: c8ad9ctoken-secret: 2e4d610cf3e7426eusage-bootstrap-authentication: "true"usage-bootstrap-signing: "true"auth-extra-groups: system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: kubelet-bootstrap
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.iokind: Groupname: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: node-autoapprove-bootstrap
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.iokind: Groupname: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: node-autoapprove-certificate-rotation
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.iokind: Groupname: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:annotations:rbac.authorization.kubernetes.io/autoupdate: "true"labels:kubernetes.io/bootstrapping: rbac-defaultsname: system:kube-apiserver-to-kubelet
rules:- apiGroups:- ""resources:- nodes/proxy- nodes/stats- nodes/log- nodes/spec- nodes/metricsverbs:- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: system:kube-apiservernamespace: ""
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:kube-apiserver-to-kubelet
subjects:- apiGroup: rbac.authorization.k8s.iokind: Username: kube-apiserver
EOFmkdir /root/coredns;cd /root/coredns
cat > coredns.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:name: corednsnamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:kubernetes.io/bootstrapping: rbac-defaultsname: system:coredns
rules:- apiGroups:- ""resources:- endpoints- services- pods- namespacesverbs:- list- watch- apiGroups:- discovery.k8s.ioresources:- endpointslicesverbs:- list- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:annotations:rbac.authorization.kubernetes.io/autoupdate: "true"labels:kubernetes.io/bootstrapping: rbac-defaultsname: system:coredns
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:coredns
subjects:
- kind: ServiceAccountname: corednsnamespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:name: corednsnamespace: kube-system
data:Corefile: |.:53 {errorshealth {lameduck 5s}readykubernetes cluster.local in-addr.arpa ip6.arpa {fallthrough in-addr.arpa ip6.arpa}prometheus :9153forward . /etc/resolv.conf {max_concurrent 1000}cache 30loopreloadloadbalance}
---
apiVersion: apps/v1
kind: Deployment
metadata:name: corednsnamespace: kube-systemlabels:k8s-app: kube-dnskubernetes.io/name: "CoreDNS"
spec:# replicas: not specified here:# 1. Default is 1.# 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on.strategy:type: RollingUpdaterollingUpdate:maxUnavailable: 1selector:matchLabels:k8s-app: kube-dnstemplate:metadata:labels:k8s-app: kube-dnsspec:priorityClassName: system-cluster-criticalserviceAccountName: corednstolerations:- key: "CriticalAddonsOnly"operator: "Exists"nodeSelector:kubernetes.io/os: linuxaffinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 100podAffinityTerm:labelSelector:matchExpressions:- key: k8s-appoperator: Invalues: ["kube-dns"]topologyKey: kubernetes.io/hostnamecontainers:- name: corednsimage: registry.cn-beijing.aliyuncs.com/dotbalo/coredns:1.8.6 imagePullPolicy: IfNotPresentresources:limits:memory: 170Mirequests:cpu: 100mmemory: 70Miargs: [ "-conf", "/etc/coredns/Corefile" ]volumeMounts:- name: config-volumemountPath: /etc/corednsreadOnly: trueports:- containerPort: 53name: dnsprotocol: UDP- containerPort: 53name: dns-tcpprotocol: TCP- containerPort: 9153name: metricsprotocol: TCPsecurityContext:allowPrivilegeEscalation: falsecapabilities:add:- NET_BIND_SERVICEdrop:- allreadOnlyRootFilesystem: truelivenessProbe:httpGet:path: /healthport: 8080scheme: HTTPinitialDelaySeconds: 60timeoutSeconds: 5successThreshold: 1failureThreshold: 5readinessProbe:httpGet:path: /readyport: 8181scheme: HTTPdnsPolicy: Defaultvolumes:- name: config-volumeconfigMap:name: corednsitems:- key: Corefilepath: Corefile
---
apiVersion: v1
kind: Service
metadata:name: kube-dnsnamespace: kube-systemannotations:prometheus.io/port: "9153"prometheus.io/scrape: "true"labels:k8s-app: kube-dnskubernetes.io/cluster-service: "true"kubernetes.io/name: "CoreDNS"
spec:selector:k8s-app: kube-dnsclusterIP: 192.168.0.2ports:- name: dnsport: 53protocol: UDP- name: dns-tcpport: 53protocol: TCP- name: metricsport: 9153protocol: TCP
EOFmkdir /root/metrics-server;cd /root/metrics-server
cat > metrics-server.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:labels:k8s-app: metrics-servername: metrics-servernamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:k8s-app: metrics-serverrbac.authorization.k8s.io/aggregate-to-admin: "true"rbac.authorization.k8s.io/aggregate-to-edit: "true"rbac.authorization.k8s.io/aggregate-to-view: "true"name: system:aggregated-metrics-reader
rules:
- apiGroups:- metrics.k8s.ioresources:- pods- nodesverbs:- get- list- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:k8s-app: metrics-servername: system:metrics-server
rules:
- apiGroups:- ""resources:- pods- nodes- nodes/stats- namespaces- configmapsverbs:- get- list- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:labels:k8s-app: metrics-servername: metrics-server-auth-readernamespace: kube-system
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccountname: metrics-servernamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:k8s-app: metrics-servername: metrics-server:system:auth-delegator
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:auth-delegator
subjects:
- kind: ServiceAccountname: metrics-servernamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:k8s-app: metrics-servername: system:metrics-server
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:metrics-server
subjects:
- kind: ServiceAccountname: metrics-servernamespace: kube-system
---
apiVersion: v1
kind: Service
metadata:labels:k8s-app: metrics-servername: metrics-servernamespace: kube-system
spec:ports:- name: httpsport: 443protocol: TCPtargetPort: httpsselector:k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:labels:k8s-app: metrics-servername: metrics-servernamespace: kube-system
spec:selector:matchLabels:k8s-app: metrics-serverstrategy:rollingUpdate:maxUnavailable: 0template:metadata:labels:k8s-app: metrics-serverspec:containers:- args:- --cert-dir=/tmp- --secure-port=4443- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname- --kubelet-use-node-status-port- --metric-resolution=15s- --kubelet-insecure-tls- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem # change to front-proxy-ca.crt for kubeadm- --requestheader-username-headers=X-Remote-User- --requestheader-group-headers=X-Remote-Group- --requestheader-extra-headers-prefix=X-Remote-Extra-image: registry.cn-beijing.aliyuncs.com/dotbalo/metrics-server:0.5.0imagePullPolicy: IfNotPresentlivenessProbe:failureThreshold: 3httpGet:path: /livezport: httpsscheme: HTTPSperiodSeconds: 10name: metrics-serverports:- containerPort: 4443name: httpsprotocol: TCPreadinessProbe:failureThreshold: 3httpGet:path: /readyzport: httpsscheme: HTTPSinitialDelaySeconds: 20periodSeconds: 10resources:requests:cpu: 100mmemory: 200MisecurityContext:readOnlyRootFilesystem: truerunAsNonRoot: truerunAsUser: 1000volumeMounts:- mountPath: /tmpname: tmp-dir- name: ca-sslmountPath: /etc/kubernetes/pkinodeSelector:kubernetes.io/os: linuxpriorityClassName: system-cluster-criticalserviceAccountName: metrics-servervolumes:- emptyDir: {}name: tmp-dir- name: ca-sslhostPath:path: /etc/kubernetes/pki---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:labels:k8s-app: metrics-servername: v1beta1.metrics.k8s.io
spec:group: metrics.k8s.iogroupPriorityMinimum: 100insecureSkipTLSVerify: trueservice:name: metrics-servernamespace: kube-systemversion: v1beta1versionPriority: 100
EOF3.相关证书生成（仅在master01操作）
master01节点下载证书生成工具
[root@k8s-master01 bootstrap]# cd /opt/src/
[root@k8s-master01 src]# mv cfssl_1.6.1_linux_amd64 cfssl
[root@k8s-master01 src]# mv cfssl-certinfo_1.6.1_linux_amd64 cfssl-certinfo
[root@k8s-master01 src]# mv cfssljson_1.6.1_linux_amd64 cfssljson
[root@k8s-master01 src]# cp cfssl* /usr/local/bin/
[root@k8s-master01 src]# chmod +x /usr/local/bin/cfssl*
[root@k8s-master01 src]# ll /usr/local/bin/cfssl*
-rwxr-xr-x. 1 root root 16659824 6月 13 02:00 /usr/local/bin/cfssl
-rwxr-xr-x. 1 root root 13502544 6月 13 02:00 /usr/local/bin/cfssl-certinfo
-rwxr-xr-x. 1 root root 11029744 6月 13 02:00 /usr/local/bin/cfssljson
3.1.生成etcd证书
特别说明除外，以下操作在所有master节点操作
3.1.1所有master节点创建证书存放目录
mkdir /etc/etcd/ssl -p3.1.2master01节点生成etcd证书
cd /root/pki/# 生成一个ca机构，给etcd认证
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca# 生成etcd证书和etcd证书的key（如果你觉得以后可能会扩容，可以在ip那多写几个预留出来）
cfssl gencert \-ca=/etc/etcd/ssl/etcd-ca.pem \-ca-key=/etc/etcd/ssl/etcd-ca-key.pem \-config=ca-config.json \-hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,10.4.7.11,10.4.7.12,10.4.7.21 \-profile=kubernetes \etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd3.1.3将证书复制到其他节点
Master='k8s-master02 k8s-master03'for NODE in $Master; do ssh $NODE "mkdir -p /etc/etcd/ssl"; for FILE in etcd-ca-key.pem etcd-ca.pem etcd-key.pem etcd.pem; do scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}; done; done3.2.生成k8s相关证书
特别说明除外，以下操作在所有master节点操作
3.2.1所有k8s节点创建证书存放目录
mkdir -p /etc/kubernetes/pki3.2.2master01节点生成k8s证书
# 生成一个根证书
cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca# 192.168.0.1是service网段的第一个地址，需要计算，10.4.7.10为高可用vip地址cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=ca-config.json \
-hostname=192.168.0.1,10.4.7.10,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,10.4.7.11,10.4.7.12,10.4.7.21,10.4.7.22,10.4.7.200 \
-profile=kubernetes apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver3.2.3生成apiserver聚合证书
# 生成一个ca机构，给后续的证书认证
cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca # 有一个警告，可以忽略cfssl gencert \
-ca=/etc/kubernetes/pki/front-proxy-ca.pem \
-ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem \
-config=ca-config.json \
-profile=kubernetes front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client3.2.4生成controller-manage的证书
cfssl gencert \-ca=/etc/kubernetes/pki/ca.pem \-ca-key=/etc/kubernetes/pki/ca-key.pem \-config=ca-config.json \-profile=kubernetes \manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager# 自定义一个k8s用户，如下叫kubernetes，此用户绑定ca.pem根证书，传递给apiserver，这样此账户-证书-apiserver进行了关联kubectl config set-cluster kubernetes \--certificate-authority=/etc/kubernetes/pki/ca.pem \--embed-certs=true \--server=https://10.4.7.10:8443 \--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig# 定义一个用户账号（system:kube-controller-manager），用这个用户账号，跟上述创建的自定义一个k8s用户kubernetes进行绑定，
用于使用提供的认证信息和命名空间将请求发送到指定的集群。给这个绑定起个名字system:kube-controller-manager@kuberneteskubectl config set-context system:kube-controller-manager@kubernetes \--cluster=kubernetes \--user=system:kube-controller-manager \--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig# 定义用于向 k8s 集群进行身份验证的客户端凭据。将客户端证书controller-manager.pem根controller-manager-key.pem
服务端证书上传进去，用于账户（system:kube-controller-manager）申请的验证kubectl config set-credentials system:kube-controller-manager \--client-certificate=/etc/kubernetes/pki/controller-manager.pem \--client-key=/etc/kubernetes/pki/controller-manager-key.pem \--embed-certs=true \--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig# 把这写信息作为承载式文件kubectl config use-context system:kube-controller-manager@kubernetes \--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig3.2.5生成scheduler的证书
cfssl gencert \-ca=/etc/kubernetes/pki/ca.pem \-ca-key=/etc/kubernetes/pki/ca-key.pem \-config=ca-config.json \-profile=kubernetes \scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/schedulerkubectl config set-cluster kubernetes \--certificate-authority=/etc/kubernetes/pki/ca.pem \--embed-certs=true \--server=https://10.4.7.10:8443 \--kubeconfig=/etc/kubernetes/scheduler.kubeconfigkubectl config set-credentials system:kube-scheduler \--client-certificate=/etc/kubernetes/pki/scheduler.pem \--client-key=/etc/kubernetes/pki/scheduler-key.pem \--embed-certs=true \--kubeconfig=/etc/kubernetes/scheduler.kubeconfigkubectl config set-context system:kube-scheduler@kubernetes \--cluster=kubernetes \--user=system:kube-scheduler \--kubeconfig=/etc/kubernetes/scheduler.kubeconfigkubectl config use-context system:kube-scheduler@kubernetes \--kubeconfig=/etc/kubernetes/scheduler.kubeconfig3.2.6生成kube-proxy的证书
cfssl gencert \-ca=/etc/kubernetes/pki/ca.pem \-ca-key=/etc/kubernetes/pki/ca-key.pem \-config=ca-config.json \-profile=kubernetes \admin-csr.json | cfssljson -bare /etc/kubernetes/pki/adminkubectl config set-cluster kubernetes \--certificate-authority=/etc/kubernetes/pki/ca.pem \--embed-certs=true \--server=https://10.4.7.10:8443 \--kubeconfig=/etc/kubernetes/admin.kubeconfigkubectl config set-credentials kubernetes-admin \--client-certificate=/etc/kubernetes/pki/admin.pem \--client-key=/etc/kubernetes/pki/admin-key.pem \--embed-certs=true \--kubeconfig=/etc/kubernetes/admin.kubeconfigkubectl config set-context kubernetes-admin@kubernetes \--cluster=kubernetes \--user=kubernetes-admin \--kubeconfig=/etc/kubernetes/admin.kubeconfigkubectl config use-context kubernetes-admin@kubernetes --kubeconfig=/etc/kubernetes/admin.kubeconfig3.2.7创建ServiceAccount Key ——secret
openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub3.2.8将证书发送到其他master节点
for NODE in k8s-master02 k8s-master03; do ssh $NODE "mkdir -p /etc/kubernetes/pki"; for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}; done; for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}; done; done3.2.9查看证书（k8s_all）
ls /etc/kubernetes/pki/
admin.csr apiserver-key.pem ca.pem front-proxy-ca.csr front-proxy-client-key.pem scheduler.csr
admin-key.pem apiserver.pem controller-manager.csr front-proxy-ca-key.pem front-proxy-client.pem scheduler-key.pem
admin.pem ca.csr controller-manager-key.pem front-proxy-ca.pem sa.key scheduler.pem
apiserver.csr ca-key.pem controller-manager.pem front-proxy-client.csr sa.pub# 一共23个就对了ls /etc/kubernetes/pki/ |wc -l
234.k8s系统组件配置
4.1 安装rsyslog
将 etcd 启动的命令、参数， 通过systemctl 进行启动，但 systemctl 启动后，日志输出怎么办？
systemctl 测试过用 StandardOutput=/tmp/services/logs/xxxx.log 根本无法输出，所以我们将systemctl 启动的程序输出的日志，通过rsyslogs转发写到固定的日志中
yum install -y rsyslogrsyslogs具体如何操作？
一、在自定义test.service文件中[Service]增加StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=your program identifier二、编辑/etc/rsyslog.d/you_file.conf文件，其内容为
if $programname == 'your program identifier' then {/data/logs/kubernetes/kube-kubelet/kubele.stdout.log
}三、重启rsyslog
sudo systemctl restart rsyslog您的程序stdout / stderr仍可通过journalctl(sudo journalctl -u your program identifier获得，但它们也将在您选择的文件中提供。四、重新你的服务
systemctl restart test五、查看/var/log/file.log 文件内容4.2.etcd配置
4.2.1master01配置
# 如果要用IPv6那么把IPv4地址修改为IPv6即可
cat > /etc/etcd/etcd.config.yml << EOF
name: 'k8s-master01'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://10.4.7.11:2380'
listen-client-urls: 'https://10.4.7.11:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://10.4.7.11:2380'
advertise-client-urls: 'https://10.4.7.11:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.4.7.11:2380,k8s-master02=https://10.4.7.12:2380,k8s-master03=https://10.4.7.21:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'client-cert-auth: truetrusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'auto-tls: true
peer-transport-security:cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'peer-client-cert-auth: truetrusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF4.2.2master02配置
# 如果要用IPv6那么把IPv4地址修改为IPv6即可
cat > /etc/etcd/etcd.config.yml << EOF
name: 'k8s-master02'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://10.4.7.12:2380'
listen-client-urls: 'https://10.4.7.12:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://10.4.7.12:2380'
advertise-client-urls: 'https://10.4.7.12:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.4.7.11:2380,k8s-master02=https://10.4.7.12:2380,k8s-master03=https://10.4.7.21:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'client-cert-auth: truetrusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'auto-tls: true
peer-transport-security:cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'peer-client-cert-auth: truetrusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF4.2.3master03配置
# 如果要用IPv6那么把IPv4地址修改为IPv6即可
cat > /etc/etcd/etcd.config.yml << EOF
name: 'k8s-master03'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://10.4.7.21:2380'
listen-client-urls: 'https://10.4.7.21:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://10.4.7.21:2380'
advertise-client-urls: 'https://10.4.7.21:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.4.7.11:2380,k8s-master02=https://10.4.7.12:2380,k8s-master03=https://10.4.7.21:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'client-cert-auth: truetrusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'auto-tls: true
peer-transport-security:cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'peer-client-cert-auth: truetrusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF4.3.创建service（所有master节点操作）
4.3.1创建etcd.service并启动
cat > /usr/lib/systemd/system/etcd.service << EOF[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=etcd[Install]
WantedBy=multi-user.target
Alias=etcd3.serviceEOFvi /etc/rsyslog.d/etcd.conf
if $programname == 'etcd' then {/data/logs/etcd/etcd.stdout.log
}4.3.2创建etcd证书目录、日志目录
mkdir /etc/kubernetes/pki/etcd
ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/mkdir -p /data/logs/etcd/
touch /data/logs/etcd/etcd.stdout.logsudo systemctl restart rsyslog
systemctl daemon-reload
systemctl enable --now etcd4.3.3查看etcd状态
如果检查后有报错，三台etcd重启后在查看
# 如果要用IPv6那么把IPv4地址修改为IPv6即可
export ETCDCTL_API=3
etcdctl --endpoints="10.4.7.11:2379,10.4.7.12:2379,10.4.7.21:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem endpoint status --write-out=table
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.0.0.83:2379 | c0c8142615b9523f | 3.5.4 | 20 kB | false | false | 2 | 9 | 9 | |
| 10.0.0.82:2379 | de8396604d2c160d | 3.5.4 | 20 kB | false | false | 2 | 9 | 9 | |
| 10.0.0.81:2379 | 33c9d6df0037ab97 | 3.5.4 | 20 kB | true | false | 2 | 9 | 9 | |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+[root@k8s-master01 pki]# 4.3.4 etcd数据定时备份
mkdir -p /data/backup/etcd
vi /etc/etcd/backup.sh
#!/bin/shcd /var/lib
name="etcd-bak"`date "+%Y%m%d"`
tar -cvf "/data/backup/etcd/"$name".tar.gz" etcd
chmod 755 /etc/etcd/backup.sh
crontab -e
00 00 * * * /etc/etcd/backup.sh或者SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
00 00 * * * (/path/to/backup.sh)
crontab -l
00 00 * * * /etc/etcd/backup.shsystemctl restart crond
5.高可用配置
5.1在lb01和lb02两台服务器上操作 (Lb)
5.1.1安装keepalived和haproxy服务
yum -y install keepalived haproxy5.1.2修改haproxy配置文件（两台配置文件一样）
# cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bakcat >/etc/haproxy/haproxy.cfg<<"EOF"
globalmaxconn 2000ulimit-n 16384log 127.0.0.1 local0 errstats timeout 30sdefaultslog globalmode httpoption httplogtimeout connect 5000timeout client 50000timeout server 50000timeout http-request 15stimeout http-keep-alive 15sfrontend monitor-inbind *:33305mode httpoption httplogmonitor-uri /monitorfrontend k8s-masterbind 0.0.0.0:8443bind 127.0.0.1:8443mode tcpoption tcplogtcp-request inspect-delay 5sdefault_backend k8s-masterbackend k8s-mastermode tcpoption tcplogoption tcp-checkbalance roundrobindefault-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100server k8s-master01 10.4.7.11:6443 checkserver k8s-master02 10.4.7.12:6443 checkserver k8s-master03 10.4.7.21:6443 check
EOF5.1.3lb01配置keepalived master节点
#cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bakcat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalivedglobal_defs {router_id LVS_DEVEL
}
vrrp_script chk_apiserver {script "/etc/keepalived/check_apiserver.sh"interval 5 weight -5fall 2rise 1
}
vrrp_instance VI_1 {state MASTERinterface eth1mcast_src_ip 10.4.7.22virtual_router_id 51priority 100nopreemptadvert_int 2authentication {auth_type PASSauth_pass K8SHA_KA_AUTH}virtual_ipaddress {10.4.7.10}track_script {chk_apiserver
} }EOF5.1.4lb02配置keepalived backup节点
# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bakcat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalivedglobal_defs {router_id LVS_DEVEL
}
vrrp_script chk_apiserver {script "/etc/keepalived/check_apiserver.sh"interval 5 weight -5fall 2rise 1}
vrrp_instance VI_1 {state BACKUPinterface eth1mcast_src_ip 10.4.7.200virtual_router_id 51priority 50nopreemptadvert_int 2authentication {auth_type PASSauth_pass K8SHA_KA_AUTH}virtual_ipaddress {10.4.7.10}track_script {chk_apiserver
} }EOF5.1.5健康检查脚本配置（两台lb主机）
cat > /etc/keepalived/check_apiserver.sh << EOF
#!/bin/basherr=0
for k in \$(seq 1 3)
docheck_code=\$(pgrep haproxy)if [[ \$check_code == "" ]]; thenerr=\$(expr \$err + 1)sleep 1continueelseerr=0breakfi
doneif [[ \$err != "0" ]]; thenecho "systemctl stop keepalived"/usr/bin/systemctl stop keepalivedexit 1
elseexit 0
fi
EOF# 给脚本授权chmod +x /etc/keepalived/check_apiserver.sh5.1.6启动服务
systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived5.1.7测试高可用
# 能ping同[root@k8s-node02 ~]# ping 10.4.7.10# 能telnet访问[root@k8s-node02 ~]# telnet 10.4.7.10 8443# 关闭主节点，看vip是否漂移到备节点6.k8s组件配置（区别于第4点）
所有k8s节点创建以下目录
mkdir -p /etc/kubernetes/manifests/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes6.1.部署 supervisord 工具
使用supervisord 工具代替systemctl
yum install -y https://mirrors.aliyun.com/epel/epel-release-latest-8.noarch.rpm
yum clean all && yum makecache
yum install -y supervisor启动supervisor
sudo supervisord -c /etc/supervisord.conf
sudo supervisorctl -c /etc/supervisord.conf写入启动脚本中
echo -e "supervisord -c /etc/supervisord.conf\nsupervisorctl -c /etc/supervisord.conf" >> /etc/rc.d/rc.local6.2.创建apiserver（所有master节点）
6.2.1master01节点配置
mkdir -p /opt/kubernetes/
mkdir -p /data/logs/kubernetes/kube-apiserver/
touch /data/logs/kubernetes/kube-apiserver/apiserver.stdout.log
vim /opt/kubernetes/kube-apiserver-startup.sh # 注意vim后:set paste
#!/bin/bash
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit/usr/local/bin/kube-apiserver \--v=2 \--logtostderr=true \--allow-privileged=true \--bind-address=0.0.0.0 \--secure-port=6443 \--advertise-address=10.4.7.11 \--service-cluster-ip-range=192.168.0.0/16 \--feature-gates=IPv6DualStack=true \--service-node-port-range=30000-32767 \--etcd-servers=https://10.4.7.11:2379,https://10.4.7.12:2379,https://10.4.7.21:2379 \--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \--etcd-certfile=/etc/etcd/ssl/etcd.pem \--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \--client-ca-file=/etc/kubernetes/pki/ca.pem \--tls-cert-file=/etc/kubernetes/pki/apiserver.pem \--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \--kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \--kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \--service-account-key-file=/etc/kubernetes/pki/sa.pub \--service-account-signing-key-file=/etc/kubernetes/pki/sa.key \--service-account-issuer=https://kubernetes.default.svc.cluster.local \--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \--authorization-mode=Node,RBAC \--enable-bootstrap-token-auth=true \--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \--requestheader-allowed-names=aggregator \--requestheader-group-headers=X-Remote-Group \--requestheader-extra-headers-prefix=X-Remote-Extra- \--requestheader-username-headers=X-Remote-User \--enable-aggregator-routing=true# --token-auth-file=/etc/kubernetes/token.csvchmod +x /opt/kubernetes/kube-apiserver-startup.sh
vim /etc/supervisord.d/kube-apiserver.ini # apiserver 使用 supervisord 启动的配置文件
[program:kube-apiserver-7-11]
command=/opt/kubernetes/kube-apiserver-startup.sh
numprocs=1
directory=/opt/kubernetes/
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false6.2.2master02节点配置
mkdir -p /opt/kubernetes/
mkdir -p /data/logs/kubernetes/kube-apiserver/
touch /data/logs/kubernetes/kube-apiserver/apiserver.stdout.log
vim /opt/kubernetes/kube-apiserver-startup.sh # 注意vim后:set paste
#!/bin/bash
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit/usr/local/bin/kube-apiserver \--v=2 \--logtostderr=true \--allow-privileged=true \--bind-address=0.0.0.0 \--secure-port=6443 \--advertise-address=10.4.7.12 \--service-cluster-ip-range=192.168.0.0/16 \--feature-gates=IPv6DualStack=true \--service-node-port-range=30000-32767 \--etcd-servers=https://10.4.7.11:2379,https://10.4.7.12:2379,https://10.4.7.21:2379 \--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \--etcd-certfile=/etc/etcd/ssl/etcd.pem \--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \--client-ca-file=/etc/kubernetes/pki/ca.pem \--tls-cert-file=/etc/kubernetes/pki/apiserver.pem \--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \--kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \--kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \--service-account-key-file=/etc/kubernetes/pki/sa.pub \--service-account-signing-key-file=/etc/kubernetes/pki/sa.key \--service-account-issuer=https://kubernetes.default.svc.cluster.local \--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \--authorization-mode=Node,RBAC \--enable-bootstrap-token-auth=true \--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \--requestheader-allowed-names=aggregator \--requestheader-group-headers=X-Remote-Group \--requestheader-extra-headers-prefix=X-Remote-Extra- \--requestheader-username-headers=X-Remote-User \--enable-aggregator-routing=true# --token-auth-file=/etc/kubernetes/token.csvchmod +x /opt/kubernetes/kube-apiserver-startup.sh
vim /etc/supervisord.d/kube-apiserver.ini # apiserver 使用 supervisord 启动的配置文件
[program:kube-apiserver-7-12]
command=/opt/kubernetes/kube-apiserver-startup.sh
numprocs=1
directory=/opt/kubernetes/
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false6.2.3master03节点配置
mkdir -p /opt/kubernetes/
mkdir -p /data/logs/kubernetes/kube-apiserver/
touch /data/logs/kubernetes/kube-apiserver/apiserver.stdout.log
vim /opt/kubernetes/kube-apiserver-startup.sh # 注意vim后:set paste
#!/bin/bash
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit/usr/local/bin/kube-apiserver \--v=2 \--logtostderr=true \--allow-privileged=true \--bind-address=0.0.0.0 \--secure-port=6443 \--advertise-address=10.4.7.12 \--service-cluster-ip-range=192.168.0.0/16 \--feature-gates=IPv6DualStack=true \--service-node-port-range=30000-32767 \--etcd-servers=https://10.4.7.11:2379,https://10.4.7.12:2379,https://10.4.7.21:2379 \--etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \--etcd-certfile=/etc/etcd/ssl/etcd.pem \--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \--client-ca-file=/etc/kubernetes/pki/ca.pem \--tls-cert-file=/etc/kubernetes/pki/apiserver.pem \--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \--kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \--kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \--service-account-key-file=/etc/kubernetes/pki/sa.pub \--service-account-signing-key-file=/etc/kubernetes/pki/sa.key \--service-account-issuer=https://kubernetes.default.svc.cluster.local \--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \--authorization-mode=Node,RBAC \--enable-bootstrap-token-auth=true \--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \--requestheader-allowed-names=aggregator \--requestheader-group-headers=X-Remote-Group \--requestheader-extra-headers-prefix=X-Remote-Extra- \--requestheader-username-headers=X-Remote-User \--enable-aggregator-routing=truechmod +x /opt/kubernetes/kube-apiserver-startup.sh
vim /etc/supervisord.d/kube-apiserver.ini # apiserver 使用 supervisord 启动的配置文件
[program:kube-apiserver-7-21]
command=/opt/kubernetes/kube-apiserver-startup.sh
numprocs=1
directory=/opt/kubernetes/
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false6.2.4启动apiserver（所有master节点）
[root@k8s-master01 ~]# supervisorctl update
[root@k8s-master02 ~]# supervisorctl update
[root@k8s-master03 ~]# supervisorctl update[root@k8s-master01 ~]# supervisorctl status
kube-apiserver-7-21 RUNNING pid 1421, uptime 16:33:20[root@k8s-master02 ~]# supervisorctl status
kube-apiserver-7-12 RUNNING pid 1989, uptime 0:36:19[root@k8s-master03 ~]# supervisorctl status
kube-apiserver-7-21 RUNNING pid 1989, uptime 0:36:11使用supervisorctl update后报错
error: <class 'FileNotFoundError'>, [Errno 2] No such file or directory: file: /usr/lib/python3.6/site-packages/supervisor/xmlrpc.py line: 560没有使用如下命令启动
sudo supervisord -c /etc/supervisord.conf
sudo supervisorctl -c /etc/supervisord.conf
6.3.配置kube-controller-manager service
# 所有master节点配置，且配置相同
# 172.7.0.0/16 为pod网段，按需求设置你自己的网段
vim /opt/kubernetes/kube-controller-manager-startup.sh # 注意vim后:set paste
#!/bin/sh
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit/usr/local/bin/kube-controller-manager \--v=2 \--logtostderr=true \--bind-address=127.0.0.1 \--root-ca-file=/etc/kubernetes/pki/ca.pem \--cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \--cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \--service-account-private-key-file=/etc/kubernetes/pki/sa.key \--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \--leader-elect=true \--use-service-account-credentials=true \--node-monitor-grace-period=40s \--node-monitor-period=5s \--pod-eviction-timeout=2m0s \--controllers=*,bootstrapsigner,tokencleaner \--allocate-node-cidrs=true \--feature-gates=IPv6DualStack=true \--service-cluster-ip-range=192.168.0.0/16 \--cluster-cidr=172.7.0.0/16 \--node-cidr-mask-size-ipv4=24 \--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pemmkdir -p /data/logs/kubernetes/kube-controller-manager
chmod u+x /opt/kubernetes/kube-controller-manager-startup.sh
touch /data/logs/kubernetes/kube-controller-manager/controller.stdout.logvim /etc/supervisord.d/kube-controller-manager.ini # controller-manager 使用 supervisord 启动的配置文件
[program:kube-controller-manager-7-11]
command=/opt/kubernetes/kube-controller-manager-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/kubernetes/ ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-controller-manager/controller.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false6.3.1启动kube-controller-manager，并查看状态
[root@k8s-master01 ~]# supervisorctl update
[root@k8s-master02 ~]# supervisorctl update
[root@k8s-master03 ~]# supervisorctl update[root@k8s-master01 ~]# supervisorctl status
kube-apiserver-7-11 RUNNING pid 2225, uptime 0:58:44
kube-controller-manager-7-11 RUNNING pid 2711, uptime 0:00:38[root@k8s-master02 ~]# supervisorctl status
kube-apiserver-7-12 RUNNING pid 1989, uptime 1:00:42
kube-controller-manager-7-12 RUNNING pid 2092, uptime 0:02:06[root@k8s-master03 ~]# supervisorctl status
kube-apiserver-7-21 RUNNING pid 1578, uptime 0:52:27
kube-controller-manager-7-21 RUNNING pid 1657, uptime 0:00:326.3.配置kube-scheduler service
6.3.1所有master节点配置，且配置相同
mkdir -p /data/logs/kubernetes/kube-scheduler
touch /data/logs/kubernetes/kube-scheduler/scheduler.stdout.logvim /opt/kubernetes/kube-scheduler-startup.sh # 注意vim后:set paste
#!/bin/sh
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit/usr/local/bin/kube-scheduler \--v=2 \--logtostderr=true \--bind-address=127.0.0.1 \--leader-elect=true \--kubeconfig=/etc/kubernetes/scheduler.kubeconfigchmod u+x /opt/kubernetes/kube-scheduler-startup.sh
vim /etc/supervisord.d/kube-scheduler.ini # controller-scheduler 使用 supervisord 启动的配置文件
[program:kube-scheduler-7-11]
command=/opt/kubernetes/kube-scheduler-startup.sh
numprocs=1
directory=/opt/kubernetes/
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=4
stdout_capture_maxbytes=1MB
stdout_events_enabled=false6.3.2启动并查看服务状态
[root@k8s-master01 ~]# supervisorctl update
[root@k8s-master02 ~]# supervisorctl update
[root@k8s-master03 ~]# supervisorctl update[root@k8s-master01 ~]# supervisorctl status
kube-apiserver-7-11 RUNNING pid 2225, uptime 2:38:21
kube-controller-manager-7-11 RUNNING pid 2711, uptime 1:40:15
kube-scheduler-7-11 RUNNING pid 2829, uptime 0:01:43[root@k8s-master02 ~]# supervisorctl status
kube-apiserver-7-12 RUNNING pid 1989, uptime 2:35:56
kube-controller-manager-7-12 RUNNING pid 2092, uptime 1:37:20
kube-scheduler-7-12 RUNNING pid 2201, uptime 0:02:00[root@k8s-master03 ~]# supervisorctl status
kube-apiserver-7-21 RUNNING pid 1578, uptime 2:27:49
kube-controller-manager-7-21 RUNNING pid 1657, uptime 1:35:54
kube-scheduler-7-21 RUNNING pid 1757, uptime 0:02:117.TLS Bootstrapping配置
7.1在master01上配置
cd /root/bootstrapkubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true --server=https://10.4.7.10:8443 \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfigkubectl config set-credentials tls-bootstrap-token-user \
--token=c8ad9c.2e4d610cf3e7426e \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfigkubectl config set-context tls-bootstrap-token-user@kubernetes \
--cluster=kubernetes \
--user=tls-bootstrap-token-user \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfigkubectl config use-context tls-bootstrap-token-user@kubernetes \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig# token的位置在bootstrap.secret.yaml，如果修改的话到这个文件修改执行 kubectl get cs 发现会报错 The connection to the server localhost:8080 was refused - did you specify the right host or port?
原因是kubectl 默认使用/root/.kube/config文件中，配置的集群角色的账户，跟api通信，config文件其实就是我们上述配置的bootstrap-kubelet.kubeconfig
mkdir -p /root/.kube ; cp /etc/kubernetes/admin.kubeconfig /root/.kube/config[root@k8s-master01 ~]# cd /root/.kube/ [root@k8s-master01 .kube]# kubectl config view apiVersion: v1 clusters: - cluster:certificate-authority-data: DATA+OMITTEDserver: https://10.4.7.10:8443name: kubernetes contexts: - context:cluster: kubernetesuser: kubernetes-adminname: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-adminuser:client-certificate-data: REDACTEDclient-key-data: REDACTED [root@k8s-master01 .kube]#
7.2查看集群状态，没问题的话继续后续操作
[root@k8s-master01 .kube]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-1 Healthy {"health":"true","reason":""}
etcd-2 Healthy {"health":"true","reason":""}
etcd-0 Healthy {"health":"true","reason":""} [root@k8s-master01 .kube]# cd /root/bootstrap/
[root@k8s-master01 bootstrap]# kubectl create -f bootstrap.secret.yaml
secret/bootstrap-token-c8ad9c created
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-certificate-rotation created
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created8.node节点配置
8.1.在master01上将证书复制到node节点
cd /etc/kubernetes/for NODE in k8s-master02 k8s-master03 k8s-node01 k8s-node02 ; do ssh $NODE mkdir -p /etc/kubernetes/pki; for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}; done; done8.2.kubelet配置
8.2.1所有k8s节点创建相关目录
mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/vi /usr/lib/systemd/system/kubelet.service # 所有k8s节点配置kubelet service，set paste
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service[Service]
ExecStart=/usr/local/bin/kubelet \--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig \--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \--config=/etc/kubernetes/kubelet-conf.yml \--container-runtime=remote \--runtime-request-timeout=15m \--container-runtime-endpoint=unix:///run/containerd/containerd.sock \--cgroup-driver=systemd \--node-labels=node.kubernetes.io/node=''
# --feature-gates=IPv6DualStack=true
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=kubelet[Install]
WantedBy=multi-user.target
vi /etc/rsyslog.d/kubelet.conf
if $programname == 'kubelet' then {/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
}mkdir -p /data/logs/kubernetes/kube-kubelet/
touch /data/logs/kubernetes/kube-kubelet/kubelet.stdout.logsudo systemctl restart rsyslog8.2.2所有k8s节点创建kubelet的配置文件
vi /etc/kubernetes/kubelet-conf.yml # set paste
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:anonymous:enabled: falsewebhook:cacheTTL: 2m0senabled: truex509:clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:mode: Webhookwebhook:cacheAuthorizedTTL: 5m0scacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 192.168.0.2
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:imagefs.available: 15%memory.available: 100Minodefs.available: 10%nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s8.2.3启动kubelet
systemctl daemon-reload
systemctl restart kubelet
systemctl enable --now kubelet8.2.4查看集群
[root@k8s-master01 rsyslog.d]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master01 NotRead <none> 108m v1.24.0
k8s-master02 NotRead <none> 10m v1.24.0
k8s-master03 NotRead <none> 7m5s v1.24.0
k8s-node01 NotRead <none> 4m8s v1.24.0
k8s-node02 NotRead <none> 11s v1.24.08.3.kube-proxy配置
8.3.1此配置只在master01操作
cp /etc/kubernetes/admin.kubeconfig /etc/kubernetes/kube-proxy.kubeconfig
8.3.2将kubeconfig发送至其他节点
for NODE in k8s-master02 k8s-master03; do scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig; donefor NODE in k8s-node01 k8s-node02 ; do scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig; done8.3.3所有k8s节点添加kube-proxy的配置和service文件
vim /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target[Service]
ExecStart=/usr/local/bin/kube-proxy \--config=/etc/kubernetes/kube-proxy.yaml \--v=2Restart=always
RestartSec=10s
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=kube-proxy[Install]
WantedBy=multi-user.targetvi /etc/rsyslog.d/kube-proxy.conf
if $programname == 'kube-proxy' then {/data/logs/kubernetes/kube-proxy/kube-proxy.stdout.log
}mkdir -p /data/logs/kubernetes/kube-proxy/
touch /data/logs/kubernetes/kube-proxy/kube-proxy.stdout.logsudo systemctl restart rsyslogvi /etc/kubernetes/kube-proxy.yaml # set paste
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:acceptContentTypes: ""burst: 10contentType: application/vnd.kubernetes.protobufkubeconfig: /etc/kubernetes/kube-proxy.kubeconfigqps: 5
clusterCIDR: 172.7.0.0/16
configSyncPeriod: 15m0s
conntrack:max: nullmaxPerCore: 32768min: 131072tcpCloseWaitTimeout: 1h0m0stcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:masqueradeAll: falsemasqueradeBit: 14minSyncPeriod: 0ssyncPeriod: 30s
ipvs:masqueradeAll: trueminSyncPeriod: 5sscheduler: "rr"syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms8.3.4启动kube-proxy
systemctl daemon-reloadsystemctl restart kube-proxysystemctl enable --now kube-proxy9. 部署件Calico
以下步骤只在master01操作
CNI网络插件最主要的功能就是实现POD资源能够跨宿主机进行通信
常见的CNI网络插件:
Flannel、Calico、Canal、Contiv、OpenContrail、NSx-T、Kube-router
9.1 Calico的讲解
https://blog.csdn.net/Jerry00713/article/details/125429199
9.2 部署Calico
为了贴合生产，我们直接模拟node节点是大于50个
下载 Kubernetes API 数据存储的 Calico 网络清单。
cd /root/ curl https://projectcalico.docs.tigera.io/manifests/calico-typha.yaml -o calico.yamlvi calico.yaml # 将CALICO_IPV4POOL_CIDR 跟对应的 Value 注释去掉，并修改cluster-cidr 值
vi calico.yaml # 在Calico的讲解中提到过，大于50node节点，名为calico-typha的Deployment资源的副本数最好为大于3，为了节省资源，使用1
最后使用以下命令应用清单，使其以 pod 的形式运行。
kubectl apply -f calico.yaml
9.3 查看容器状态
一开始会显示如下，在创建中
等待很长一段时间，等待都running，其中会出现
[root@k8s-master01 ~]# kubectl get pod -A
Unable to connect to the server: net/http: TLS handshake timeout[root@k8s-master01 ~]# kubectl get pod -A
Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get pods)[root@k8s-master01 ~]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6766647d54-46mpn 0/1 ContainerCreating 0 11s
kube-system calico-node-9dzc7 0/1 Init:0/2 0 12s
kube-system calico-node-b9jbw 0/1 Init:0/2 0 12s
kube-system calico-node-dhq2c 0/1 Init:0/2 0 12s
kube-system calico-node-l9hwj 0/1 Init:0/2 0 12s
kube-system calico-node-zm9n7 0/1 Init:0/2 0 12s
kube-system calico-typha-6c589979d4-rr8cn 0/1 ContainerCreating 0 12s
[root@k8s-master01 ~]# 会发现其实Calico 启动一个calico-kube-controllers 的pod。启动一个calico-typha的pod，这是由于我们设置的副本数为1。最后是每个节点启动一个pod，也说明了Calico把每个节点当作一个路由进行2层数据的转发
[root@k8s-master01 ~]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-56cdb7c587-8stfx 1/1 Running 6 (10m ago) 39m
kube-system calico-node-62ndd 1/1 Running 1 (9m18s ago) 39m
kube-system calico-node-ch624 1/1 Running 0 39m
kube-system calico-node-kgtk7 1/1 Running 0 39m
kube-system calico-node-p55wj 1/1 Running 4 (8m29s ago) 39m
kube-system calico-node-rhjdq 1/1 Running 1 (9m18s ago) 39m
kube-system calico-typha-6775694657-jd66w 1/1 Running 1 (22m ago) 39m[root@k8s-master01 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master01 Ready <none> 8d v1.24.0
k8s-master02 Ready <none> 8d v1.24.0
k8s-master03 Ready <none> 8d v1.24.0
k8s-node01 Ready <none> 8d v1.24.0
k8s-node02 Ready <none> 8d v1.24.010.安装CoreDNS
10.1以下步骤只在master01操作
10.1.1修改文件
[root@k8s-master01 ~]# cd /root/coredns
[root@k8s-master01 coredns]# vi coredns.yaml # 将 clusterIP: 改成你自己的coredns的IP，我的IP是192.168.0.2 ，注意coredns的IP一定要跟集群的第一个地址区分，我的集群第一个地址是192.168.0.1（也就是配置apiserver证书的第一个地址）。
[root@k8s-master01 coredns]# cat coredns.yaml | grep clusterIP:clusterIP: 192.168.0.210.1.2安装
[root@k8s-master01 coredns]# kubectl create -f coredns.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created11.安装Metrics Server
k8s各个组件将自己的健康状态输送给了Metrics 接口，以便比如 Promethe 监控软件进行监控
11.1以下步骤只在master01操作
11.1.1安装Metrics-server
在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率
# 安装metrics server
[root@k8s-master01 ~]# cd /root/metrics-server/
[root@k8s-master01 metrics-server]# kubectl apply -f metrics-server.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created11.1.2稍等片刻查看状态
[root@k8s-master01 metrics-server]# kubectl top node
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)稍等片刻查看状态[root@k8s-master01 metrics-server]# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master01 475m 47% 1233Mi 66%
k8s-master02 557m 55% 1120Mi 60%
k8s-master03 350m 35% 1100Mi 59%
k8s-node01 103m 10% 636Mi 34%
k8s-node02 992m 99% 713Mi 38% 12.集群验证
12.1部署pod资源
执行后会卡住，是正常的，因为此命令是写yaml文件并应用，其中本地没有busybox:1.28镜像还要从公网下载
cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:name: busyboxnamespace: default
spec:containers:- name: busyboximage: busybox:1.28command:- sleep- "3600"imagePullPolicy: IfNotPresentrestartPolicy: Always
EOF# 查看kubectl get pod
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 17s
12.2用pod解析默认命名空间中的kubernetes
[root@k8s-master01 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 192.168.0.1 <none> 443/TCP 9d[root@k8s-master01 metrics-server]# kubectl exec busybox -n default -- nslookup kubernetes
Server: 192.168.0.2
Address 1: 192.168.0.2 kube-dns.kube-system.svc.cluster.localName: kubernetes
Address 1: 192.168.0.1 kubernetes.default.svc.cluster.local
[root@k8s-master01 metrics-server]# 12.3测试跨命名空间是否可以解析
[root@k8s-master01 ~]# kubectl exec busybox -n default -- nslookup kube-dns.kube-system
Server: 192.168.0.2
Address 1: 192.168.0.2 kube-dns.kube-system.svc.cluster.localName: kube-dns.kube-system
Address 1: 192.168.0.2 kube-dns.kube-system.svc.cluster.local
12.4每个节点都必须要能访问Kubernetes的kubernetes svc 443和kube-dns的service 53
[root@k8s-master01 ~]# telnet 192.168.0.1 443
Trying 192.168.0.1...
Connected to 192.168.0.1.
Escape character is '^]'.[root@k8s-master01 ~]# telnet 192.168.0.2 53
Trying 192.168.0.2...
Connected to 192.168.0.2.
Escape character is '^]'.[root@k8s-master01 ~]# curl 192.168.0.2:53
curl: (52) Empty reply from server12.5Pod和Pod之间要能通
[root@k8s-master01 ~]# kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 0 53m 172.7.122.129 k8s-master02 <none> <none>[root@k8s-master01 ~]# kubectl get pod -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-56cdb7c587-8stfx 1/1 Running 7 (7m32s ago) 3h8m 10.88.0.2 k8s-master03 <none> <none>
calico-node-62ndd 1/1 Running 1 (157m ago) 3h8m 10.4.7.200 k8s-node02 <none> <none>
calico-node-ch624 1/1 Running 0 3h8m 10.4.7.11 k8s-master01 <none> <none>
calico-node-kgtk7 1/1 Running 0 3h8m 10.4.7.12 k8s-master02 <none> <none>
calico-node-p55wj 1/1 Running 4 (157m ago) 3h8m 10.4.7.22 k8s-node01 <none> <none>
calico-node-rhjdq 1/1 Running 1 (157m ago) 3h8m 10.4.7.21 k8s-master03 <none> <none>
calico-typha-6775694657-jd66w 1/1 Running 1 (171m ago) 3h8m 10.4.7.22 k8s-node01 <none> <none>
coredns-6d86b45487-x5zwx 1/1 Running 0 3m7s 172.7.32.130 k8s-master01 <none> <none>
metrics-server-6d9df85947-5hssb 1/1 Running 0 59m 172.7.58.193 k8s-node02 <none> <none># 进入busybox ping其他节点上的pod
[root@k8s-master01 ~]# kubectl exec -ti busybox -- sh
/ # ping 172.7.32.130
PING 172.7.32.130 (172.7.32.130): 56 data bytes
64 bytes from 172.7.32.130: seq=0 ttl=62 time=4.384 ms
64 bytes from 172.7.32.130: seq=1 ttl=62 time=6.536 ms
64 bytes from 172.7.32.130: seq=2 ttl=62 time=9.680 ms# 可以连通证明这个pod是可以跨命名空间和跨主机通信的12.6创建三个副本，可以看到3个副本分布在不同的节点上（用完可以删了）
cat > nginx.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:name: nginx-deploymentlabels:app: nginx
spec:replicas: 3selector:matchLabels:app: nginxtemplate:metadata:labels:app: nginxspec:containers:- name: nginximage: nginx:1.14.2ports:- containerPort: 80EOF[root@k8s-master01 ~]# kubectl apply -f nginx.yaml
deployment.apps/nginx-deployment created[root@k8s-master01 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 70 (49m ago) 2d22h
nginx-deployment-6595874d85-9565p 1/1 Running 1 (10m ago) 14m
nginx-deployment-6595874d85-m5rmh 1/1 Running 0 14m
nginx-deployment-6595874d85-sd9fr 1/1 Running 0 14m# 删除 nginx
[root@k8s-master01 ~]# kubectl delete -f nginx.yaml
deployment.apps "nginx-deployment" deleted13.安装dashboard
13.1 下载dashboard_v2.4.0安装包
官网：GitHub - kubernetes/dashboard at v2.4.0
找到 recommended.yaml 文件
13.2 部署dashboard_v2.4.0
[root@k8s-master01 ~]# mkdir /root/dashboard;cd /root/dashboard
[root@k8s-master01 dashboard]# rz recommended.yaml
[root@k8s-master01 dashboard]# vi recommended.yaml # 按照如下修改
原配置文件为：
修改后：
kind: Service
apiVersion: v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard
spec:type: NodePortports:- port: 443nodePort: 31245targetPort: 8443selector:k8s-app: kubernetes-dashboard---
应用资源配置清单
[root@k8s-master01 dashboard]# kubectl apply -f recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
查看所属node以及端口
kubectl -n kubernetes-dashboard get pod,svc -o wide使用浏览器访问任意一个节点的31245端口，就可以访问dashboard页面，说明了，dashboard的页面是https://dashboard_IP:443，而我们通过NodePort类型，将31245端口映射到了dashboard容器中的443，而NodePort代表的是宿主机，node节点，所以访问node节点的IP+31245，就是访问的dashboard容器中的443端口（https://dashboard_IP:443）。注意，必须使用稍微低版本的火狐浏览器，如果使用谷歌等，会发现页面一直显示空白。是因为这个高内核的浏览器，无法容忍没有安全认证的https协议的网页，所以如果要显示，还需要在配置ssl，请看这篇文章
配置从k8s中配置出一个token，让我们有权限登录到dashboard，所以我需要创建一个admin-user用户，绑定一个k8s集群角色，然后通过admin-user用户生成一个token
[root@k8s-master01 dashboard]# vi create-admin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:name: admin-usernamespace: kubernetes-dashboard---apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: admin-user
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: cluster-admin
subjects:
- kind: ServiceAccountname: admin-usernamespace: kubernetes-dashboard
应用资源配置清单
[root@k8s-master01 dashboard]# kubectl apply -f create-admin.yaml
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created[root@k8s-master01 dashboard]# kubectl get serviceaccount -n kubernetes-dashboard -o yaml
apiVersion: v1
items:
- apiVersion: v1kind: ServiceAccountmetadata:annotations:kubectl.kubernetes.io/last-applied-configuration: |{"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"admin-user","namespace":"kubernetes-dashboard"}}creationTimestamp: "2022-07-12T05:13:50Z"name: admin-usernamespace: kubernetes-dashboardresourceVersion: "115033"uid: 6d3d9c79-b692-4a9f-a7a4-f569fad6c21f
- apiVersion: v1kind: ServiceAccountmetadata:creationTimestamp: "2022-07-11T23:54:08Z"name: defaultnamespace: kubernetes-dashboardresourceVersion: "72877"uid: 5436446c-aa69-424a-9289-eca0a42feeda
- apiVersion: v1kind: ServiceAccountmetadata:annotations:kubectl.kubernetes.io/last-applied-configuration: |{"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"labels":{"k8s-app":"kubernetes-dashboard"},"name":"kubernetes-dashboard","namespace":"kubernetes-dashboard"}}creationTimestamp: "2022-07-11T23:54:08Z"labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboardresourceVersion: "72879"uid: 3e559071-9bcf-43cb-9b6d-10e61a270eb3
kind: List
metadata:resourceVersion: ""创建token
[root@k8s-master01 dashboard]# kubectl -n kubernetes-dashboard create token admin-user
eyJhbGciOiJSUzI1NiIsImtpZCI6IjBua1VscDM0V1JJT3ZxRnNBdUFZdm5fd19XeDhvOHY4M25wOGxOcEdmUWsifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjU2NDg4ODc1LCJpYXQiOjE2NTY0ODUyNzUsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJhZG1pbi11c2VyIiwidWlkIjoiYjhlYzQ2N2EtYzRhZC00OTZjLWIyYTctN2YyYTNiYzdjOWFiIn19LCJuYmYiOjE2NTY0ODUyNzUsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDphZG1pbi11c2VyIn0.Ywh2TzMV3wGknts0WAVIUBhPXX-pOllI1EI_QHDSTiTNB-ii0a4iC6b8VtZaw1l3tznu3a7nO53sAUILhyhocnEUSLV7bXhWQK_kafIOTWxTOZneVi-4IA7wxrLxv06m7Xhh8somR9lfiMU4Tscnmu2_imT0aoItk3t6mAwY9umyc5PfyeCO4vjFre_ddG6B2IxVEnnzvHS8ViIOrWMQuEUYrB58RL1h7EA1AbGe0Wt2rJRGj0aZ23xf7JGTWS1slQnrjxCU_8YWXTYm0nJQWeqGqzEQbZXdKrf_aIq5ogZejPfylMOeZwXYVno_kx3NAIkjZGKtFt_--o8xyWySpg复制token到dashboard页面登录
14.ingress安装
14.1写入配置文件，并执行
原理：ingress-controller 就是把k8s内部的ningx调度器，ingress-controller跟宿主机共享网络空间，通过nodeport 80 把宿主机的80端口映射到ingress-controller的80端口，这时候配置ingress资源，比如 tomcat.od.com，其实也就是访问的是tomcat.od.com:80，这时候我们通过hosts或者named将，tomcat.od.com解析成ingress-controller的pod所在的宿主机IP，也就是访问的是宿主机的IP:80，由于访问宿主机的IP:80，也就是ingress-controller的pod的80，所以访问tomcat.od.com流量引入到了ingress-controller，ingress-controller跟api通讯获得有一个ingress资源是tomcat.od.com，这个ingress资源把流量给到了他指定的service资源，service在通过kube-proxy负载给到pod。
注：
1、在上述说的，我需要一个把访问tomcat.od.com给到一个宿主机，这个宿主机是谁，依赖于ingress-controller的pod在那个机器，所以我i们通过node选择器，固定在那些node节点运行
2、一定提前把镜像下载下来，否则会发现，ingress功能不好使，在被选定的node节点运行，此文章我们选择了使用daemonset类型，通过node选择器固定到master01、master02crictl pull willdockerhub/ingress-nginx-controller:v1.1.0 crictl registry.cn-hangzhou.aliyuncs.com/chenby/kube-webhook-certgen:v1.1.1
[root@k8s-master01 ~]# mkdir ingress;cd ingress/
[root@k8s-master01 ingress]# vim deploy.yaml
apiVersion: v1
kind: Namespace
metadata:name: ingress-nginxlabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginx---
# Source: ingress-nginx/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:labels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: ingress-nginxnamespace: ingress-nginx
automountServiceAccountToken: true
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:labels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: ingress-nginx-controllernamespace: ingress-nginx
data:allow-snippet-annotations: 'true'
---
# Source: ingress-nginx/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmname: ingress-nginx
rules:- apiGroups:- ''resources:- configmaps- endpoints- nodes- pods- secrets- namespacesverbs:- list- watch- apiGroups:- ''resources:- nodesverbs:- get- apiGroups:- ''resources:- servicesverbs:- get- list- watch- apiGroups:- networking.k8s.ioresources:- ingressesverbs:- get- list- watch- apiGroups:- ''resources:- eventsverbs:- create- patch- apiGroups:- networking.k8s.ioresources:- ingresses/statusverbs:- update- apiGroups:- networking.k8s.ioresources:- ingressclassesverbs:- get- list- watch
---
# Source: ingress-nginx/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmname: ingress-nginx
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: ingress-nginx
subjects:- kind: ServiceAccountname: ingress-nginxnamespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:labels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: ingress-nginxnamespace: ingress-nginx
rules:- apiGroups:- ''resources:- namespacesverbs:- get- apiGroups:- ''resources:- configmaps- pods- secrets- endpointsverbs:- get- list- watch- apiGroups:- ''resources:- servicesverbs:- get- list- watch- apiGroups:- networking.k8s.ioresources:- ingressesverbs:- get- list- watch- apiGroups:- networking.k8s.ioresources:- ingresses/statusverbs:- update- apiGroups:- networking.k8s.ioresources:- ingressclassesverbs:- get- list- watch- apiGroups:- ''resources:- configmapsresourceNames:- ingress-controller-leaderverbs:- get- update- apiGroups:- ''resources:- configmapsverbs:- create- apiGroups:- ''resources:- eventsverbs:- create- patch
---
# Source: ingress-nginx/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:labels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: ingress-nginxnamespace: ingress-nginx
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: ingress-nginx
subjects:- kind: ServiceAccountname: ingress-nginxnamespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-service-webhook.yaml
apiVersion: v1
kind: Service
metadata:labels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: ingress-nginx-controller-admissionnamespace: ingress-nginx
spec:type: ClusterIPports:- name: https-webhookport: 443targetPort: webhookappProtocol: httpsselector:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:annotations:labels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: ingress-nginx-controllernamespace: ingress-nginx
spec:type: NodePortexternalTrafficPolicy: LocalipFamilyPolicy: SingleStackipFamilies:- IPv4ports:- name: httpport: 80protocol: TCPtargetPort: httpappProtocol: http- name: httpsport: 443protocol: TCPtargetPort: httpsappProtocol: httpsselector:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/component: controller
---
apiVersion: apps/v1
kind: DaemonSet
metadata:labels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: ingress-nginx-controllernamespace: ingress-nginx
spec:selector:matchLabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/component: controllerrevisionHistoryLimit: 10minReadySeconds: 0template:metadata:labels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/component: controllerspec:hostNetwork: truednsPolicy: ClusterFirstWithHostNetnodeSelector:app: ingresscontainers:- name: controllerimage: willdockerhub/ingress-nginx-controller:v1.1.0 imagePullPolicy: IfNotPresentlifecycle:preStop:exec:command:- /wait-shutdownargs:- /nginx-ingress-controller- --election-id=ingress-controller-leader- --controller-class=k8s.io/ingress-nginx- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller- --validating-webhook=:8443- --validating-webhook-certificate=/usr/local/certificates/cert- --validating-webhook-key=/usr/local/certificates/keysecurityContext:capabilities:drop:- ALLadd:- NET_BIND_SERVICErunAsUser: 101allowPrivilegeEscalation: trueenv:- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: LD_PRELOADvalue: /usr/local/lib/libmimalloc.solivenessProbe:failureThreshold: 5httpGet:path: /healthzport: 10254scheme: HTTPinitialDelaySeconds: 10periodSeconds: 10successThreshold: 1timeoutSeconds: 1readinessProbe:failureThreshold: 3httpGet:path: /healthzport: 10254scheme: HTTPinitialDelaySeconds: 10periodSeconds: 10successThreshold: 1timeoutSeconds: 1ports:- name: httpcontainerPort: 80protocol: TCP- name: httpscontainerPort: 443protocol: TCP- name: webhookcontainerPort: 8443protocol: TCPvolumeMounts:- name: webhook-certmountPath: /usr/local/certificates/readOnly: trueresources:requests:cpu: 100mmemory: 90MiserviceAccountName: ingress-nginxterminationGracePeriodSeconds: 300volumes:- name: webhook-certsecret:secretName: ingress-nginx-admission
---
# Source: ingress-nginx/templates/controller-ingressclass.yaml
# We don't support namespaced ingressClass yet
# So a ClusterRole and a ClusterRoleBinding is required
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:labels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: nginxnamespace: ingress-nginx
spec:controller: k8s.io/ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
# before changing this value, check the required kubernetes version
# https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:labels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: admission-webhookname: ingress-nginx-admission
webhooks:- name: validate.nginx.ingress.kubernetes.iomatchPolicy: Equivalentrules:- apiGroups:- networking.k8s.ioapiVersions:- v1operations:- CREATE- UPDATEresources:- ingressesfailurePolicy: FailsideEffects: NoneadmissionReviewVersions:- v1clientConfig:service:namespace: ingress-nginxname: ingress-nginx-controller-admissionpath: /networking/v1/ingresses
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:name: ingress-nginx-admissionnamespace: ingress-nginxannotations:helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgradehelm.sh/hook-delete-policy: before-hook-creation,hook-succeededlabels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: admission-webhook
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: ingress-nginx-admissionannotations:helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgradehelm.sh/hook-delete-policy: before-hook-creation,hook-succeededlabels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: admission-webhook
rules:- apiGroups:- admissionregistration.k8s.ioresources:- validatingwebhookconfigurationsverbs:- get- update
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: ingress-nginx-admissionannotations:helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgradehelm.sh/hook-delete-policy: before-hook-creation,hook-succeededlabels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: admission-webhook
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: ingress-nginx-admission
subjects:- kind: ServiceAccountname: ingress-nginx-admissionnamespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:name: ingress-nginx-admissionnamespace: ingress-nginxannotations:helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgradehelm.sh/hook-delete-policy: before-hook-creation,hook-succeededlabels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: admission-webhook
rules:- apiGroups:- ''resources:- secretsverbs:- get- create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:name: ingress-nginx-admissionnamespace: ingress-nginxannotations:helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgradehelm.sh/hook-delete-policy: before-hook-creation,hook-succeededlabels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: admission-webhook
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: ingress-nginx-admission
subjects:- kind: ServiceAccountname: ingress-nginx-admissionnamespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:name: ingress-nginx-admission-createnamespace: ingress-nginxannotations:helm.sh/hook: pre-install,pre-upgradehelm.sh/hook-delete-policy: before-hook-creation,hook-succeededlabels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: admission-webhook
spec:template:metadata:name: ingress-nginx-admission-createlabels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: admission-webhookspec:containers:- name: createimage: registry.cn-hangzhou.aliyuncs.com/chenby/kube-webhook-certgen:v1.1.1 imagePullPolicy: IfNotPresentargs:- create- --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc- --namespace=$(POD_NAMESPACE)- --secret-name=ingress-nginx-admissionenv:- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespacesecurityContext:allowPrivilegeEscalation: falserestartPolicy: OnFailureserviceAccountName: ingress-nginx-admissionnodeSelector:kubernetes.io/os: linuxsecurityContext:runAsNonRoot: truerunAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:name: ingress-nginx-admission-patchnamespace: ingress-nginxannotations:helm.sh/hook: post-install,post-upgradehelm.sh/hook-delete-policy: before-hook-creation,hook-succeededlabels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: admission-webhook
spec:template:metadata:name: ingress-nginx-admission-patchlabels:helm.sh/chart: ingress-nginx-4.0.10app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/version: 1.1.0app.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: admission-webhookspec:containers:- name: patchimage: registry.cn-hangzhou.aliyuncs.com/chenby/kube-webhook-certgen:v1.1.1 imagePullPolicy: IfNotPresentargs:- patch- --webhook-name=ingress-nginx-admission- --namespace=$(POD_NAMESPACE)- --patch-mutating=false- --secret-name=ingress-nginx-admission- --patch-failure-policy=Failenv:- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespacesecurityContext:allowPrivilegeEscalation: falserestartPolicy: OnFailureserviceAccountName: ingress-nginx-admissionnodeSelector:kubernetes.io/os: linuxsecurityContext:runAsNonRoot: truerunAsUser: 2000
给node01 node02 添加label，使其ingress-controller指定跑在这两个节点
[root@k8s-master01 ingress]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master01 Ready <none> 19h v1.24.0
k8s-master02 Ready <none> 15h v1.24.0
k8s-master03 Ready <none> 19h v1.24.0
k8s-node01 Ready <none> 19h v1.24.0
k8s-node02 Ready <none> 19h v1.24.0
[root@k8s-master01 ingress]# kubectl label node k8s-node01 k8s-node02 app=ingress
node/k8s-node01 labeled
node/k8s-node02 labeled
[root@k8s-master01 ingress]# kubectl get label --show-lablekubectl get label --show-lable^C
[root@k8s-master01 ingress]# kubectl get node -A --show-labels
NAME STATUS ROLES AGE VERSION LABELS
k8s-master01 Ready <none> 19h v1.24.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master01,kubernetes.io/os=linux,node.kubernetes.io/node=
k8s-master02 Ready <none> 15h v1.24.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master02,kubernetes.io/os=linux,node.kubernetes.io/node=
k8s-master03 Ready <none> 19h v1.24.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master03,kubernetes.io/os=linux,node.kubernetes.io/node=
k8s-node01 Ready <none> 19h v1.24.0 app=ingress,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node01,kubernetes.io/os=linux,node.kubernetes.io/node=
k8s-node02 Ready <none> 19h v1.24.0 app=ingress,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node02,kubernetes.io/os=linux,node.kubernetes.io/node=
[root@k8s-master01 ingress]# 应用资源清单
[root@k8s-master01 ingress]# kubectl apply -f deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
daemonset.apps/ingress-nginx-controller created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created查看pod运行状态
[root@k8s-master01 ingress]# kubectl get pod -n ingress-nginx -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-admission-create-hqrb7 0/1 Completed 0 14h 172.7.122.130 k8s-master02 <none> <none>
ingress-nginx-admission-patch-bbkj4 0/1 Completed 2 14h 172.7.58.193 k8s-node02 <none> <none>
ingress-nginx-controller-dzvz4 1/1 Running 0 116s 10.4.7.200 k8s-node02 <none> <none>
ingress-nginx-controller-qmg6n 1/1 Running 0 37m 10.4.7.22 k8s-node01 <none> <none>14.2 配置反代
思路：目前已经部署了两个 ingress-controller 的 pod，分别在 node01 node02 ，其中ingress-controller 的 pod 跟宿主机共享网络名称空间，而ingress-controller启动的是80端口，所以只要把想要访问网站流量送入node01 node02 宿主机的80端口，就能够被ingress-controller接收到，后续就是k8s内部匹配ingressg规则转发给service资源的事情了。目前我们有两个 ingress-controller，所以完全可以让访问的流量通过LB负载均衡机制，分别访问两个ingress-controller。我们使用nginx作为负载均衡机制。
制作过程：我们找其中一个节点，比如是node02，安装nginx，在nginx中配置只要这个节点接收到是访问 *.od.com 这个域名的网站，全部转发给10.4.7.22:80 和 10.4.7.200:80，然后在named中做域名解析，只要是访问*.od.com ，统统抛给node02。但如果node02挂了怎么办，为了保证高可用，node01 node02 分别安装nginx，都配置一样的nginx反向代理配置，并安装ksspalived，使其实现如果node02挂了，可以飘逸到node01 。在第五章的时候，已经在node01 node02配置高可用，VIP为10.4.7.10
node01 node02 操作：
yum install -y nginx
vi /etc/nginx/conf.d/od.com.conf
server {server_name *.od.com;location / {proxy_pass http://default_backend_nginx;proxy_set_header Host $http_host;proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;}
}upstream default_backend_nginx {# 使用的是daemonset，所有的nodes都放到upstream中server 10.4.7.22:80 max_fails=3 fail_timeout=10s;server 10.4.7.200:80 max_fails=3 fail_timeout=10s;
}systemctl restart nginx ;systemctl enable nginx
14.3 安装测试应用
[root@k8s-master01 ~]# mkdir/root/nginx; cd /root/nginx
[root@k8s-master01 nginx]# vi deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata: name: nginx-dpnamespace: defaultlabels:app: nginx-lable
spec:selector:matchLabels:app: nginx-1.7.9replicas: 1template: metadata:labels:app: nginx-1.7.9spec:containers:- name: nginximage: nginx:1.7.9ports:- containerPort: 80[root@k8s-master01 nginx]# vi svc.yaml
apiVersion: v1
kind: Service
metadata:name: nginx-svcnamespace: defaultlabels:app: nginx-svc-label
spec:selector:app: nginx-1.7.9ports:- port: 881protocol: TCPtargetPort: 80[root@k8s-master01 nginx]# vi ingress.yaml
kind: Ingress
apiVersion: networking.k8s.io/v1
metadata: name: nginx-ingressnamespace: defaultlabels:app: nginx-ingress-label
spec:ingressClassName: nginxrules:- host: nginx.od.comhttp:paths:- pathType: Prefixpath: "/"backend:service:name: nginx-svcport:number: 881应用资源清单
[root@k8s-master01 nginx]# kubectl apply -f deploy.yaml
deployment.apps/nginx-dp created
[root@k8s-master01 nginx]# kubectl apply -f svc.yaml
service/nginx-svc created
[root@k8s-master01 nginx]# kubectl apply -f ingress.yaml
ingress.networking.k8s.io/nginx-ingress created[root@k8s-master01 nginx]# kubectl get all -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-dp-5d9dbd4c6d-zfv42 1/1 Running 0 14m 172.7.195.3 k8s-master03 <none> <none>NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 192.168.0.1 <none> 443/TCP 23h <none>
service/nginx-svc ClusterIP 192.168.139.174 <none> 881/TCP 13m app=nginx-1.7.9NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/nginx-dp 1/1 1 1 14m nginx nginx:1.7.9 app=nginx-1.7.9NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/nginx-dp-5d9dbd4c6d 1 1 1 14m nginx nginx:1.7.9 app=nginx-1.7.9,pod-template-hash=5d9dbd4c6d验证nginx的pod跟service无问题
# 直接访问pod的80端口[root@k8s-master01 nginx]# curl 172.7.195.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>body {width: 35em;margin: 0 auto;font-family: Tahoma, Verdana, Arial, sans-serif;}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html># 直接访问service的881端口，因为service 配置的是 881 代理pod的 80[root@k8s-master01 nginx]# curl 192.168.139.174:881
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>body {width: 35em;margin: 0 auto;font-family: Tahoma, Verdana, Arial, sans-serif;}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
配置named，让访问nginx.od.com 的流量抛给10.4.7.10这个VIP，通过nginx的配置把流量引入到node的80端口
[root@k8s-master01 nginx]# vi /var/named/od.com.zone
[root@k8s-master01 nginx]# vi /var/named/od.com.zone
$ORIGIN od.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.od.com. dnsadmin.od.com. (2020010501 ; serial10800 ; refresh (3 hours)900 ; retry (15 minutes)604800 ; expire (1 week)86400 ; minimum (1 day))NS dns.od.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
nginx A 10.4.7.10[root@k8s-master01 nginx]# systemctl restart named
[root@k8s-master01 nginx]# dig -t A nginx.od.com @10.4.7.11 +short
10.4.7.10
验证访问curl nginx.od.com
[root@k8s-master01 nginx]# curl nginx.od.com
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>body {width: 35em;margin: 0 auto;font-family: Tahoma, Verdana, Arial, sans-serif;}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@k8s-master01 nginx]# 14.3 DefaultBackend 配置
介绍：没有 rules 的 Ingress 将所有流量发送到同一个默认后端。 defaultBackend 通常是ingress控制器的配置选项，而非在 Ingress 资源中指定。如果 hosts 或 paths 都没有与 Ingress 对象中的 HTTP 请求匹配，则流量将路由到默认后端
理解：说白了意思就是一个请求的url，被ingress-controller接收到后，如果没有ingress规则匹配上，默认走defaultBackend的规则，自己测试了一下，有和没有DefaultBackend这个配置，无任何影响。具体为什么呢？比如有一个ingress资源匹配的是nginx.od.com，但访问nginx.od.com/123，是不是就匹配了defaultBackend呢？并不是。defaultBackend 只管理那些完全没有任何规则匹配上资源，比如nihao.od.com或者nihao.od.com/123，在ingress中根本连nihao.od.com资源都没有的。所以功能鸡肋，目前没发现任何价值。再就是我们本身都通过named域名解析，才能转发给ingress-controller，如果named没配置，根本转发不到ingress-controller。而且在配置中还发现了
[root@k8s-master01 backend]# mkdir /root/backend; cd /root/backend
[root@k8s-master01 backend]# vi deploy_backend.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: default-http-backendlabels:app: default-http-backend
spec:replicas: 1selector:matchLabels:app: default-http-backendtemplate:metadata:labels:app: default-http-backendspec:terminationGracePeriodSeconds: 60containers:- name: default-http-backend# Any image is permissible as long as:# 1. It serves a 404 page at /# 2. It serves 200 on a /healthz endpointimage: registry.cn-hangzhou.aliyuncs.com/chenby/defaultbackend-amd64:1.5imagePullPolicy: IfNotPresentlivenessProbe:httpGet:path: /healthzport: 8080scheme: HTTPinitialDelaySeconds: 30timeoutSeconds: 5ports:- containerPort: 8080resources:limits:cpu: 10mmemory: 20Mirequests:cpu: 10mmemory: 20Mivi svc.yaml
apiVersion: v1
kind: Service
metadata:name: default-http-backendlabels:app: default-http-backend
spec:ports:- port: 80targetPort: 8080selector:app: default-http-backend注意：既然是没有规则的url，所以必须的有ingress资源，但是怎么配置的。rules规则些什么的，默认给出的defaultBackend参数，指向default-http-backend，但是无语的是，如果就按照如下create -f 应用到k8s后，根本不好用。此参数还必须搭配在其他的ingress资源中使用，真实鸡肋
apiVersion: networking.k8s.io/v1 kind: Ingress metadata:name: tomcat-demo spec:defaultBackend:service:name: default-http-backendport: number: 80
[root@k8s-master01 ~]# cd /root/nginx
[root@k8s-master01 nginx]# vi ingress.yaml # 修改nginx的资源配置清单
kind: Ingress
apiVersion: networking.k8s.io/v1
metadata: name: nginx-ingressnamespace: defaultlabels:app: nginx-ingress-label
spec:defaultBackend:service:name: default-http-backendport: number: 80ingressClassName: nginxrules:- host: nginx.od.comhttp:paths:- pathType: Prefixpath: "/"backend:service:name: nginx-svcport:number: 881
[root@k8s-master01 nginx]# 应用资源清单
[root@k8s-master01 backend]# kubectl apply -f deploy_backend.yaml
deployment.apps/default-http-backend created[root@k8s-master01 backend]# kubectl apply -f svc_backend.yaml
service.apps/default-http-backend created[root@k8s-master01 nginx]# kubectl apply -f ingress.yaml
ingress.networking.k8s.io/nginx-ingress changed[root@k8s-master01 nginx]# vi /var/named/od.com.zone
[root@k8s-master01 nginx]# vi /var/named/od.com.zone
$ORIGIN od.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.od.com. dnsadmin.od.com. (2020010501 ; serial10800 ; refresh (3 hours)900 ; retry (15 minutes)604800 ; expire (1 week)86400 ; minimum (1 day))NS dns.od.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
nginx A 10.4.7.10
nihao A 10.4.7.10[root@k8s-master01 nginx]# systemctl restart named
[root@k8s-master01 nginx]# dig -t A nihao.od.com @10.4.7.11 +short
10.4.7.10
验证
[root@k8s-master01 nginx]# curl nginx.od.com/123
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.7.9</center>
</body>
</html>[root@k8s-master01 nginx]# curl nihao.od.com
default backend - 404[root@k8s-master01 nginx]# 这个才是default backend的404网页[root@k8s-master01 nginx]# curl nihao.od.com/123
default backend - 404[root@k8s-master01 nginx]#[root@k8s-master01 nginx]# curl helloword.od.com
curl: (6) Could not resolve host: helloword.od.com若遇到Ingress-nginx-controller创建yaml文件报x509: certificate signed by unknown authority等错误，查看https://blog.csdn.net/Jerry00713/article/details/125743837
二进制安装Kubernetes（k8s） v1.24.0 IPv4相关推荐
- 二进制安装Kubernetes（k8s）IPv4/IPv6双栈 v1.24.0
二进制安装Kubernetes(k8s) v1.24.0 IPv4/IPv6双栈 介绍 kubernetes二进制安装 1.23.3 和 1.23.4 和 1.23.5 和 1.23.6 和 1.24 ...
- Centos containerd和kubernetes v 1.24.0部署
containerd和kubernetes v 1.24.0 install method 1.24版本包含2项重要变更: 1,移除Dockershim 2,默认关闭各Beta APIs Centos ...
- 二进制安装Kubernetes（k8s） v1.25.0 IPv4/IPv6双栈
二进制安装Kubernetes(k8s) v1.25.0 IPv4/IPv6双栈 Kubernetes 开源不易,帮忙点个star,谢谢了
- 二进制安装Kubernetes（k8s） v1.24.3 IPv4/IPv6双栈
二进制安装Kubernetes(k8s) v1.24.3 IPv4/IPv6双栈 Kubernetes 开源不易,帮忙点个star,谢谢了
- 二进制安装Kubernetes（k8s） v1.22.10 IPv4/IPv6双栈
二进制安装Kubernetes(k8s) v1.22.10 IPv4/IPv6双栈 Kubernetes 开源不易,帮忙点个star,谢谢了
- 二进制安装Kubernetes（k8s） v1.23.7 IPv4/IPv6双栈
二进制安装Kubernetes(k8s) v1.23.7 IPv4/IPv6双栈 Kubernetes 开源不易,帮忙点个star,谢谢了
- 二进制安装Kubernetes（k8s） v1.21.13 IPv4/IPv6双栈
二进制安装Kubernetes(k8s) v1.21.13 IPv4/IPv6双栈 Kubernetes 开源不易,帮忙点个star,谢谢了
- 二进制安装Kubernetes（k8s） v1.23.6
二进制安装Kubernetes(k8s) v1.23.6 背景 kubernetes二进制安装 1.23.3 和 1.23.4 和 1.23.5 和 1.23.6 文档以及安装包已生成. 后续尽可能第 ...
- 二进制安装k8s v1.25.4 IPv4/IPv6双栈
二进制安装k8s v1.25.4 IPv4/IPv6双栈 https://github.com/cby-chen/Kubernetes 开源不易,帮忙点个star,谢谢了 介绍 kubernetes( ...
最新文章
- 雅虎某分站的XSS导致雅虎邮箱沦陷
- 动画体系知识梳理(1) 转场动画 ContentTransition 理论篇
- 自定义UISegmentedControl
- TCP/IP 建立连接的过程
- 数据在java中加时间,如何在Or.jFr.DATA时间中设置RealTimeMead时期的Java数据
- 进程间的通信方式（一）：共享内存
- 面向对象（上）知识点
- EF学习杂记39：如何重置Relationships
- 小G的项链（Manacher）
- 2014_beijing_onsite
- qt 添加依赖库lib_在QT中添加LIB的方法
- CVPR 2020 中的群组活动识别
- 后序遍历（非递归）☆
- 【转】Windows消息投递流程：WM_COMMAND消息流程
- ajax_demo:GET POST发送数据
- Phoenix=HBase+SQL,让HBase插上了翅膀
- 1's Complement和2's Complement的区别
- St. Luke’s University Health Network是世界首批试用远程患者管理解决方案Masimo SafetyNet™来协助COVID-19住院患者的机构之一
- C语言程序设计-跳马问题
- android qq 邮箱格式,QQ邮箱的正确格式有哪些？文件夹怎么发送呢
热门文章
- 弹性云服务器（Elastic Cloud Server，ECS）
- 干货！Flask 动态展示 Pyecharts 图表数据的几种方法！
- Linux中ifconfig的使用
- Solving environment: failed ResolvePackageNotFound: - ld_impl_linux-64==2.34=h53a641e_4
- 实战案例汇总，Java架构师实战视频教程
- 在Ubuntu PHP中基于wxsqlite3扩展使用加密的Sqlite数据库
- 吴恩达机器学习MATLAB代码笔记（1）梯度下降
- MySQL联合索引原理解析
- 12333提交显示服务器异常,掌上12333显示没有收到异地协助认证书什么原因_具体解决办法流程_3DM手游...
- JavaWeb开发 —— Web入门