k8s高可用集群

1 使用pacemaker搭建k8s的高可用(haproxy的高可用)
- 安装并配置haproxy
- 安装并配置pacemaker
2 k8s集群部署
- master准备
- 三个结点关闭交换分区
- 安装docker及kubelet
- 初始化集群
- 添加fence

1 使用pacemaker搭建k8s的高可用(haproxy的高可用)

server5、server6配置仓库

[root@server5 ~]# vim /etc/yum.repos.d/dvd.repo
[dvd]
name=dvd
baseurl=http://172.25.14.250/rhel7.6
gpgcheck=0[HighAvailability]
name=HighAvailability
baseurl=http://172.25.14.250/rhel7.6//addons/HighAvailability
gpgcheck=0

安装并配置haproxy

yum install -y haproxy
cd /etc/haproxy/
vim haproxy.cfg
systemctl restart haproxy.service

安装并配置pacemaker

安装并设置开机自启

yum install -y pacemaker pcs psmisc policycoreutils-python
systemctl enable --now pcsd.service

修改密码并认证

passwd hacluster
pcs cluster auth
pcs cluster auth server5 server6

集群组建

pcs cluster setup --name mycluster server5 server6

设置开机自启动集群

pcs property set stonith-enabled=false
pcs cluster start --all
pcs cluster enable --all
crm_verify -L -V
pcs  status

配置vip资源

pcs  resource create vip ocf:heartbeat:IPaddr2 ip=172.25.14.100 op monitor interval=30s
pcs  status

配置haproxy服务资源

pcs resource create haproxy systemd:haproxy op monitor interval=60s
pcs  status

资源放到一个组

pcs resource group add hagroup vip haproxy
pcs  status

2 k8s集群部署

将server1仓库的认证传给server7、server8、server9，方便后续下载镜像

master准备

server7、server8、server9做k8s master结点

三个结点关闭交换分区

[root@server7 ~]# swapoff -a
[root@server7 ~]# vim /etc/fstab
#/dev/mapper/rhel-swap   swap                    swap    defaults        0 0

安装docker及kubelet

server7、server8、server9安装docker、kubelet并启用

[root@server7 ~]# yum install -y docker-ce[root@server7 ~]# tar zxf kubeadm-1.21.3.tar.gz
[root@server7 ~]# cd packages/
[root@server7 packages]# yum install -y *[root@server7 ~]# systemctl enable --now kubelet.service
[root@server7 ~]# systemctl enable docker.service

修改文件

vim  /etc/docker/daemon.json
{"registry-mirrors": ["https://reg.westos.org"],"exec-opts": ["native.cgroupdriver=systemd"],"log-driver": "json-file","log-opts": {"max-size": "100m"},"storage-driver": "overlay2","storage-opts": ["overlay2.override_kernel_check=true"]
}vim /etc/sysctl.d/docker.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1重启服务
systemctl restart docker
sysctl --system
查看三个结点
docker info

初始化集群

修改初始化文件

[root@server7 ~]# kubeadm config print init-defaults > kubeadm-init.yaml  ##生成init文件[root@server7 ~]# vim kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:- system:bootstrappers:kubeadm:default-node-tokentoken: abcdef.0123456789abcdefttl: 24h0m0susages:- signing- authentication
kind: InitConfiguration
localAPIEndpoint:advertiseAddress: 172.25.14.7bindPort: 6443
nodeRegistration:criSocket: /var/run/dockershim.sockname: server7taints: null
---
apiServer:timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "172.25.14.100:6443"
controllerManager: {}
dns:type: CoreDNS
etcd:local:dataDir: /var/lib/etcd
imageRepository: reg.westos.org/k8s
kind: ClusterConfiguration
kubernetesVersion: 1.21.3
networking:dnsDomain: cluster.localpodSubnet: 10.244.0.0/16serviceSubnet: 10.96.0.0/12
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

k8s初始化

[root@server7 ~]# kubeadm init --config kubeadm-init.yaml --upload-certs

初始化成功

将结点加入k8s master

 kubeadm join 172.25.14.100:6443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:09c95026b52089ea481d22d82e9abff6555c7b54d3d2767c2f309b5182870360 \--control-plane --certificate-key 084495de2520d67a636265923634dfa0eb4103f8cbd2df4963a948178f14e69f

将结点加入k8s node

kubeadm join 172.25.14.100:6443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:09c95026b52089ea481d22d82e9abff6555c7b54d3d2767c2f309b5182870360

安装网络组件(flannel)

[root@server7 ~]# vim kube-flannel.yml
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:name: psp.flannel.unprivilegedannotations:seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/defaultseccomp.security.alpha.kubernetes.io/defaultProfileName: docker/defaultapparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/defaultapparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:privileged: falsevolumes:- configMap- secret- emptyDir- hostPathallowedHostPaths:- pathPrefix: "/etc/cni/net.d"- pathPrefix: "/etc/kube-flannel"- pathPrefix: "/run/flannel"readOnlyRootFilesystem: false# Users and groupsrunAsUser:rule: RunAsAnysupplementalGroups:rule: RunAsAnyfsGroup:rule: RunAsAny# Privilege EscalationallowPrivilegeEscalation: falsedefaultAllowPrivilegeEscalation: false# CapabilitiesallowedCapabilities: ['NET_ADMIN', 'NET_RAW']defaultAddCapabilities: []requiredDropCapabilities: []# Host namespaceshostPID: falsehostIPC: falsehostNetwork: truehostPorts:- min: 0max: 65535# SELinuxseLinux:# SELinux is unused in CaaSPrule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: flannel
rules:
- apiGroups: ['extensions']resources: ['podsecuritypolicies']verbs: ['use']resourceNames: ['psp.flannel.unprivileged']
- apiGroups:- ""resources:- podsverbs:- get
- apiGroups:- ""resources:- nodesverbs:- list- watch
- apiGroups:- ""resources:- nodes/statusverbs:- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: flannel
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: flannel
subjects:
- kind: ServiceAccountname: flannelnamespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:name: flannelnamespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:name: kube-flannel-cfgnamespace: kube-systemlabels:tier: nodeapp: flannel
data:cni-conf.json: |{"name": "cbr0","cniVersion": "0.3.1","plugins": [{"type": "flannel","delegate": {"hairpinMode": true,"isDefaultGateway": true}},{"type": "portmap","capabilities": {"portMappings": true}}]}net-conf.json: |{"Network": "10.244.0.0/16","Backend": {"Type": "host-gw"}}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:name: kube-flannel-dsnamespace: kube-systemlabels:tier: nodeapp: flannel
spec:selector:matchLabels:app: flanneltemplate:metadata:labels:tier: nodeapp: flannelspec:affinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/osoperator: Invalues:- linuxhostNetwork: truepriorityClassName: system-node-criticaltolerations:- operator: Existseffect: NoScheduleserviceAccountName: flannelinitContainers:- name: install-cniimage: k8s/flannel:v0.14.0command:- cpargs:- -f- /etc/kube-flannel/cni-conf.json- /etc/cni/net.d/10-flannel.conflistvolumeMounts:- name: cnimountPath: /etc/cni/net.d- name: flannel-cfgmountPath: /etc/kube-flannel/containers:- name: kube-flannelimage: k8s/flannel:v0.14.0command:- /opt/bin/flanneldargs:- --ip-masq- --kube-subnet-mgrresources:requests:cpu: "100m"memory: "50Mi"limits:cpu: "100m"memory: "50Mi"securityContext:privileged: falsecapabilities:add: ["NET_ADMIN", "NET_RAW"]env:- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespacevolumeMounts:- name: runmountPath: /run/flannel- name: flannel-cfgmountPath: /etc/kube-flannel/volumes:- name: runhostPath:path: /run/flannel- name: cnihostPath:path: /etc/cni/net.d- name: flannel-cfgconfigMap:name: kube-flannel-cfg[root@server7 ~]# kubectl apply -f kube-flannel.yml

查看

再创建server10，重复上述过程，作为node结点加入k8s集群

此时master端查看

此时server10

[root@server10 packages]# mkdir -p /etc/cni/net.d
[root@server10 packages]# cd /etc/cni/net.d/
[root@server10 net.d]# vim 10-flannel.conflist
{"name": "cbr0","cniVersion": "0.3.1","plugins": [{"type": "flannel","delegate": {"hairpinMode": true,"isDefaultGateway": true}},{"type": "portmap","capabilities": {"portMappings": true}}]
}
[root@server10 net.d]# systemctl daemon-reload
[root@server10 net.d]# systemctl restart kubelet

master再次查看，server10已经ready

server7运行镜像

[root@server7 ~]# kubectl run demo --image=myapp:v1
[root@server7 ~]# kubectl get pod -o wide
[root@server7 ~]# curl IP

添加fence

server5、server6查看

pcs status

真机

[root@foundation14 kiosk]# cd /etc/cluster/
[root@foundation14 cluster]# scp fence_xvm.key server6:/etc/cluster/

server5 server6安装

yum install -y fence-virt

创建fence

[root@server5 ~]# pcs stonith create vmfence fence_xvm pcmk_host_map="server5:server5;server6:server6" op monitor interval=60s
[root@server5 ~]# pcs status

开启fence

[root@server5 ~]# pcs property set stonith-enabled=true

企业实战-Kubernetes(十四)k8s高可用集群相关推荐

部署一套完整的K8s高可用集群（二进制-V1.20）
<部署一套完整的企业级K8s集群> v1.20,二进制方式作者信息李振良(阿良),微信:xyz12366699 DevOps实战学院 http://www.aliangedu.cn 说 ...
ansible-playbook部署K8S高可用集群
通过ansible-playbook,以Kubeadm方式部署K8S高可用集群(多主多从). kubernetes安装目录: /etc/kubernetes/KubeConfig: ~/.kube/c ...
k8s高可用集群搭建部署
简介 k8s普通搭建出来只是单master节点,如果该节点挂掉,则整个集群都无法调度,K8s高可用集群是用多个master节点加负载均衡节点组成,外层再接高可用分布式存储集群例如ceph集群,实现计算 ...
自动化运维之k8s——Helm、普罗米修斯、EFK日志管理、k8s高可用集群(未完待续)
一.k8s高可用集群(3.12日课) 几种常见的集群结构 1.堆叠的 etcd 拓扑 2. 外部 etcd 拓扑 3.外部 etcd 拓扑(load balancer = lvs + keepaliv ...
Kubernetes — 使用 kubeadm 部署高可用集群
目录文章目录目录 Kubernetes 在生产环境中架构高可用集群部署拓扑 1.网络代理配置 2.Load Balancer 环境准备 3.Kubernetes Cluster 环境准备安装 ...
kubernets(四)创建高可用集群
Kubernetes(一)认识 kubernetes Kubernets(二)部署非高可用Kubernetes集群的环境准备 Kubernets(三)部署非高可用Kubernetes集群-通过阿里云源 ...
K8S高可用集群架构部署 dashborad插件部署 Nginx实现动静分离 K8S在线升级
K8S官方文档注意:该集群每个master节点都默认由kubeadm生成了etcd容器,组成etcd集群.正常使用集群,etcd的集群不能超过一半为down状态. docker的namespace: ...
k8s高可用集群多个主节点_部署高可用k8s集群
高可用集群指 1个lb + 3个master(etcd) + n个node,生产环境都推荐这种安装方式新版的k8s,etcd节点已经可以完美和master节点共存于同一台服务器上: etcd有3种方式 ...
运维企业专题（11）RHCS高可用集群下MySql数据库与共享磁盘（单点写入、多点写入）的设置
实验环境主机名 IP 服务 server1 172.25.6.1 ricci,luci, iscsi,mysql-server server2 172.25.6.2 ricci,iscsi,mysq ...

企业实战-Kubernetes(十四)k8s高可用集群