linux下离线安装k8s集群1.19.4附带nfs存储（kubeadm方式）
linux下离线安装k8s集群1.19.4附带nfs存储
活动地址：毕业季·进击的技术er
一，环境简介
|
kubernetes-1.19.4集群部署计划 |
||||
|
序号 |
服务器配置 |
IP地址 |
操作系统 |
备注 |
|
1 |
cpu：2c 内存：4G |
192.168.217.16 |
centos 7.6 |
k8s主节点 |
|
2 |
cpu：2c |
192.168.217.17 |
centos 7.6 |
k8s从节点 |
|
3 |
cpu：2c |
192.168.217.18 |
centos 7.6 |
k8s从节点 |
三台服务器均为虚拟机，网络配置为nat模式。
链接：https://pan.baidu.com/s/19PTj1VwpvaSxYlhbFuqP6w?pwd=k8ss
提取码：k8ss
离线安装包的链接！！！！！！！！！！！！！包含docker环境
二，
关于域名映射问题和网络问题，主机名称修改如下，如何修改在此不讨论。
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.217.16 master
192.168.217.17 slave1
192.168.217.18 slave2
编辑hosts文件，域名映射如上。因是nat网络模式，因此，三台服务器自组网，三个网卡配置文件内容大体如下:
TYPE="Ethernet"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
NAME="ens33"
UUID="d4876b9f-42d8-446c-b0ae-546e812bc954"
DEVICE="ens33"
ONBOOT="yes"
IPADDR=192.168.217.16
NETMASK=255.255.255.0
GATEWAY=192.168.217.2
DNS1=192.168.217.16
三，
network服务已启用则关闭NetworkManager防止冲突
systemctl stop NetworkManager && systemctl disable NetworkManager四，时间服务器
时间服务器的搭建见本人博客：Linux ntp时间服务器的搭建和配置_zsk_john的博客-CSDN博客_linux ntp服务器搭建Linux ntp时间服务器的搭建和配置date +"%Z %z"查看ntp服务器与上层ntp的状态【命令】ntpq -premote：本机和上层ntp的ip或主机名，“+”表示优先，“*”表示次优先refid：参考上一层ntp主机地址st：stratum阶层when：多少秒前曾经同步过时间poll：下次更新在多少秒后reach：已经向上层ntp服务器要求更新的次数delay：网络延迟offset：时间补偿jit...https://blog.csdn.net/alwaysbefine/article/details/109055169
五，
内核参数修改，
这个步骤是必须要有的，k8s在安装和使用的过程中会检测这三个参数，三台服务器都要做：
vim /etc/sysctl.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
写入这几个参数在sysctl.conf 文件内，然后sysctl -p 命令 使之生效。（特别注意，这个命令之前需要执行开启ipvs内核命令 ：modprobe br_netfilter）
六，
三台服务器都关闭防火墙，selinux，swap挂载，升级内核版本到5.1
关闭防火墙命令是：
systemctl disable firewalld && systemctl stop firewalldselinux 临时关闭：setenforce 0
selinux 永久关闭：修改 /etc/selinux/config 这个文件，SELINUX=disabled
swap卸载，见本人博客：KVM虚拟机管理工作二（虚拟机磁盘优化，Centos进入dracut模式，报 /dev/centos/swap does not exist,如何恢复）_zsk_john的博客-CSDN博客_kvm虚拟机磁盘缩容量前言：KVM虚拟机的安装其实不是一个简单的事情，为什么要这么说呢？因为，KVM虚拟机在安装完毕后，我们可能会有很多定制化的需求，比如，更改虚拟机的root密码，安装一些常用软件，或者常用的软件环境。也会有扩容，缩容，增加逻辑盘以及打快照等等扩展需求。那么，KVM虚拟机的操作系统安装一般是什么要求呢？我想，第一，是需要最小化安装，这里最小化安装是为了降低KVM镜像的大小，使得镜像轻量化。第二，是关闭swap，因为很多环境是不能有swap的，相对于生产服务器来说，通常swap都是一个鸡肋的存在，并且https://blog.csdn.net/alwaysbefine/article/details/124831650这里是一个比较容易忽略的地方，卸载swap建议最好按照我的博客所写进行，否则会重新启动不了服务器。
七，升级内核
升级内核的原因是k8s运行在高版本内核下比较稳定，升级内核方法如下：
rpm -ivh kernel-ml-5.16.9-1.el7.elrepo.x86_64.rpmgrub2-set-default "CentOS Linux (5.16.9-1.el7.elrepo.x86_64) 7 (Core)"grub2-editenv list ## 查看内核启动项
六七步骤建议都完成后，统一重启服务器。三个节点都做。
八，
服务器之间的免密互信操作
具体操作见本人博客：科普扫盲---ssh免密登陆（ssh的一些小秘密）_zsk_john的博客-CSDN博客_ssh免密登录配置ssh协议和tcp/ip 协议一样非常的重要，那么，如何使用这个ssh呢？这个协议到底有什么用处呢？一，ssh协议是什么ssh是secure SHell的简写，意思为安全的shell，中文也叫安全的外壳协议（是不是比较喜感的一个名称？），那，既然都是shell了，自然是有shell的那些特征啦。SSH 主要由三部分组成：传输层协议 [SSH-TRANS]提供了服务器认证，保密性及完整性。此外它有时还提供压缩功能。 SSH-TRANS 通常运行在TCP/IP连接上，也可能用于其它可https://blog.csdn.net/alwaysbefine/article/details/123451448
九，docker环境的部署
docker离线环境部署见本人博客 ：docker的离线安装以及本地化配置_zsk_john的博客-CSDN博客docker的离线安装以及本地化配置首先需要说明离线安装的适用场景：项目具有私有云，内外网分离，安全性要求比较高，相对在线安装，离线安装优势很大，方便，灵活，不需要配置yum源以及考虑整体网络环境的事情了，只需要关心局域网的网络就可以了。废话不多说了，离线安装有RPM包安装方式，二进制安装包方式，源码编译安装方式，最为简便的方式为二进制安装包方式（没有rpm依赖问题，预编译和编译中，缺少依赖而失败的问题）。一，下...https://blog.csdn.net/alwaysbefine/article/details/110310112
这里需要说明一下，docker的版本是ce19. 03.9，该版本是和k8s的1.19.4版本适配的
正式开始部署k8s集群
一，
k8s集群规划，因此需要在环境变量内设定一个新变量，变量写在 /etc/profile 文件内，（三个服务器都要写）变量内容如下：
export no_proxy=localhost,127.0.0.1,dev.cnn,192.168.217.16,default.svc.cluster.local,svc.cluster.local,cluster.local,10.96.0.1,10.96.0.0/12,10.244.0.0/16二，
k8s基本组件的安装：
k8s-1.19.4-offline这个文件夹内的k8s.tar.gz文件解压，然后将该解压目录挂载为本地仓库。
k8s-1.19.4-offline这个文件夹内的conntrack.tar.gz解压，然后执行命令 rpm -ivh * 安装，这个是k8s的强依赖。
检查本地仓库无误后执行以下命令进行安装：
yum install -y kubeadm-1.19.4 kubelet-1.19.4 kubectl-1.19.4将服务加入自启，三个节点都要执行：
systemctl enable kubelet &&systemctl start kubelet服务状态为绿色表示服务正常：
[root@master opt]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node AgentLoaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)Drop-In: /usr/lib/systemd/system/kubelet.service.d└─10-kubeadm.confActive: active (running) since Fri 2022-07-01 18:52:58 CST; 5h 44min agoDocs: https://kubernetes.io/docs/Main PID: 1091 (kubelet)Memory: 152.8MCGroup: /system.slice/kubelet.service└─1091 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --h...Jul 01 18:59:45 master kubelet[1091]: I0701 18:59:45.844741 1091 topology_manager.go:233] [topologymanager] Topology Admit Handler
Jul 01 18:59:45 master kubelet[1091]: I0701 18:59:45.889281 1091 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "tmp-volume" (UniqueName:...
Jul 01 18:59:45 master kubelet[1091]: I0701 18:59:45.889367 1091 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "tmp-volume" (UniqueName:...
Jul 01 18:59:45 master kubelet[1091]: I0701 18:59:45.889414 1091 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "kubernetes-dashboard-tok...
Jul 01 18:59:45 master kubelet[1091]: I0701 18:59:45.889451 1091 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "kubernetes-dashboard-cer...
Jul 01 18:59:45 master kubelet[1091]: I0701 18:59:45.889488 1091 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "kubernetes-dashboard-tok...
Jul 01 18:59:46 master kubelet[1091]: W0701 18:59:46.669731 1091 pod_container_deletor.go:79] Container "3b5bb41530363d16e2478900afd45d91dbe5f9260cf8d0ac398a8d29da0a...s containers
Jul 01 18:59:46 master kubelet[1091]: W0701 18:59:46.674402 1091 pod_container_deletor.go:79] Container "3c46b2fa0a198e044fdd27507e17a14944dcee9f657be06d1e5812b16383...s containers
Jul 01 23:37:01 master kubelet[1091]: I0701 23:37:01.462295 1091 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 07ee94a447d5bed0408914de8...4794cb7ae2d9
Jul 01 23:37:01 master kubelet[1091]: I0701 23:37:01.462890 1091 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: a5996702878a2fac2c793c22b...1b5fb16772e6
Hint: Some lines were ellipsized, use -l to show in full.
三，
镜像的导入：
k8s-1.19.4-offline这个文件夹内的master-images.tar.gz在16服务器解压，然后执行批量导入命令：for i in `ls master-images`;do docker load <$i;done
k8s-1.19.4-offline这个文件夹内的slave1-images.tar.gz在17服务器解压，然后执行批量导入命令：for i in `ls slave1-images`;do docker load <$i;done
k8s-1.19.4-offline这个文件夹内的slave2-images.tar.gz在18服务器解压，然后执行批量导入命令：for i in `ls slave2-images`;do docker load <$i;done
四，
k8s-1.19.4-offline这个文件夹内的kubeadm.zip在三个服务器都解压，然后，将可执行文件kubeadm-1.19.3移动到 /usr/bin/目录下，改名为kubeadm
修改kubeadm.conf 文件，重点修改如下内容：
localAPIEndpoint:advertiseAddress: 192.168.217.16bindPort: 6443
nodeRegistration:criSocket: /var/run/dockershim.sockname: zsk.cnntaints:- effect: NoSchedulekey: node-role.kubernetes.io/master
五，
集群初始化，执行以下命令即可：
kubeadm init --config kubeadm.conf如果初始化失败的话，可以使用命令 kubeadm reset 命令进行重置，不建议删除相关环境文件重做初始化，加入节点命令在此命令的末尾，复制该命令后在其它节点运行即可加入节点，不需要对此命令进行任何改动，如果加入集群失败，可同样使用kubeadm reset 命令重新恢复环境，再次加入。
注意，此命令是在master节点执行，命令成功执行完成后，输出有节点加入命令，复制节点加入命令，在其余两个节点执行即可。
此时的集群状态应该是noready，在主节点执行命令：kubectl apply -f kube-flannel.yml 集群状态即可恢复正常。
六，
安装kubernetes-dashboard（此操作只在master节点执行，其余两个节点不执行）
dashboard.yml文件的内容如下：
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.apiVersion: v1
kind: Namespace
metadata:name: kubernetes-dashboard---apiVersion: v1
kind: ServiceAccount
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard---kind: Service
apiVersion: v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard
spec:ports:- port: 443targetPort: 8443selector:k8s-app: kubernetes-dashboard---apiVersion: v1
kind: Secret
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboard-certsnamespace: kubernetes-dashboard
type: Opaque---apiVersion: v1
kind: Secret
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboard-csrfnamespace: kubernetes-dashboard
type: Opaque
data:csrf: ""---apiVersion: v1
kind: Secret
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboard-key-holdernamespace: kubernetes-dashboard
type: Opaque---kind: ConfigMap
apiVersion: v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboard-settingsnamespace: kubernetes-dashboard---kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard
rules:# Allow Dashboard to get, update and delete Dashboard exclusive secrets.- apiGroups: [""]resources: ["secrets"]resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]verbs: ["get", "update", "delete"]# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.- apiGroups: [""]resources: ["configmaps"]resourceNames: ["kubernetes-dashboard-settings"]verbs: ["get", "update"]# Allow Dashboard to get metrics.- apiGroups: [""]resources: ["services"]resourceNames: ["heapster", "dashboard-metrics-scraper"]verbs: ["proxy"]- apiGroups: [""]resources: ["services/proxy"]resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]verbs: ["get"]---kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboard
rules:# Allow Metrics Scraper to get metrics from the Metrics server- apiGroups: ["metrics.k8s.io"]resources: ["pods", "nodes"]verbs: ["get", "list", "watch"]---apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: kubernetes-dashboard
subjects:- kind: ServiceAccountname: kubernetes-dashboardnamespace: kubernetes-dashboard---apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: kubernetes-dashboard
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: kubernetes-dashboard
subjects:- kind: ServiceAccountname: kubernetes-dashboardnamespace: kubernetes-dashboard---kind: Deployment
apiVersion: apps/v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard
spec:replicas: 1revisionHistoryLimit: 10selector:matchLabels:k8s-app: kubernetes-dashboardtemplate:metadata:labels:k8s-app: kubernetes-dashboardspec:containers:- name: kubernetes-dashboardimage: kubernetesui/dashboard:v2.0.4ports:- containerPort: 8443protocol: TCPargs:- --auto-generate-certificates- --namespace=kubernetes-dashboard# Uncomment the following line to manually specify Kubernetes API server Host# If not specified, Dashboard will attempt to auto discover the API server and connect# to it. Uncomment only if the default does not work.# - --apiserver-host=http://my-address:portvolumeMounts:- name: kubernetes-dashboard-certsmountPath: /certs# Create on-disk volume to store exec logs- mountPath: /tmpname: tmp-volumelivenessProbe:httpGet:scheme: HTTPSpath: /port: 8443initialDelaySeconds: 30timeoutSeconds: 30securityContext:allowPrivilegeEscalation: falsereadOnlyRootFilesystem: truerunAsUser: 1001runAsGroup: 2001volumes:- name: kubernetes-dashboard-certssecret:secretName: kubernetes-dashboard-certs- name: tmp-volumeemptyDir: {}serviceAccountName: kubernetes-dashboardnodeSelector:"kubernetes.io/os": linux# Comment the following tolerations if Dashboard must not be deployed on mastertolerations:- key: node-role.kubernetes.io/mastereffect: NoSchedule---kind: Service
apiVersion: v1
metadata:labels:k8s-app: dashboard-metrics-scrapername: dashboard-metrics-scrapernamespace: kubernetes-dashboard
spec:ports:- port: 8000targetPort: 8000selector:k8s-app: dashboard-metrics-scraper---kind: Deployment
apiVersion: apps/v1
metadata:labels:k8s-app: dashboard-metrics-scrapername: dashboard-metrics-scrapernamespace: kubernetes-dashboard
spec:replicas: 1revisionHistoryLimit: 10selector:matchLabels:k8s-app: dashboard-metrics-scrapertemplate:metadata:labels:k8s-app: dashboard-metrics-scraperannotations:seccomp.security.alpha.kubernetes.io/pod: 'runtime/default'spec:containers:- name: dashboard-metrics-scraperimage: kubernetesui/metrics-scraper:v1.0.6ports:- containerPort: 8000protocol: TCPlivenessProbe:httpGet:scheme: HTTPpath: /port: 8000initialDelaySeconds: 30timeoutSeconds: 30volumeMounts:- mountPath: /tmpname: tmp-volumesecurityContext:allowPrivilegeEscalation: falsereadOnlyRootFilesystem: truerunAsUser: 1001runAsGroup: 2001serviceAccountName: kubernetes-dashboardnodeSelector:"kubernetes.io/os": linux# Comment the following tolerations if Dashboard must not be deployed on mastertolerations:- key: node-role.kubernetes.io/mastereffect: NoSchedulevolumes:- name: tmp-volumeemptyDir: {}mkdir /etc/kubernetes/pki/dashboard/cd /etc/kubernetes/pki/dashboard/openssl genrsa -out tls.key 2048openssl req -new -key tls.key -subj "/CN=zsk.cnn" -out tls.csropenssl x509 -req -days 3650 -in tls.csr -CA ../ca.crt -CAkey ../ca.key -CAcreateserial -out tls.crtkubectl create secret generic kubernetes-dashboard-certs --from-file=/etc/kubernetes/pki/dashboard/ -n kube-system#执行kubectl 命令安装dashboardkubectl apply -f dashboard.yml 输出如下：
secret/kubernetes-dashboard-certs created
#集群角色绑定
kubectl create clusterrolebinding default --clusterrole=cluster-admin --serviceaccount=kube-system:default --namespace=kube-system输出如下为正确：
clusterrolebinding.rbac.authorization.k8s.io/default created
七，
安装ingress
vim ingress-nginx-values.yamlcontroller:name: controllerimage:repository: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controllertag: "v0.50.0"pullPolicy: IfNotPresentconfig:map-hash-bucket-size: "1024"proxy-body-size: "100m"ssl-protocols: "TLSv1.2 TLSv1.3"enable-modsecurity: "true"enable-owasp-modsecurity-crs: "true"error-log-level: "warn"modsecurity:config:enabled: truednsPolicy: ClusterFirstWithHostNethostNetwork: truehostPort:enabled: trueports:http: 80https: 443kind: DaemonSetresources:limits:cpu: 200mmemory: 512Mirequests:cpu: 100mmemory: 200Mi
defaultBackend:enabled: trueimage:repository: registry.cn-hangzhou.aliyuncs.com/google_containers/defaultbackendtag: "1.4"pullPolicy: IfNotPresent
此时，解压k8s-1.19.4-offline\helms这个目录下的ingress-nginx-3.25.0.tgz，上面的配置文件和压缩包同一目录下，然后执行以下命令。这里特别注意，helm这个文件先需要放到环境变量里哦，也就是移动helm 这个文件到 /usr/localbin/目录下即可。
helm install ingress-nginx -f ingress-nginx-values.yaml ingress-nginx -n ingress-nginx --create-namespaceingress用到的镜像包是：
[root@master YAML]# docker images --digests
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller <none> sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a 435df390f367 16 months ago 279MB特别注意，需要查看digest是否是3dd开始的，如果不是，需要修改它的digest，ingress依赖的是两个镜像，这两个镜像都放在了ingress目录下。
八，
部署dashboard-ingress
vim dash-ingress.yaml
这个文件的hosts需要指定，这里我用的是dash.master.com 这个域名 ，hosts里一会要写哦
|
kind: Ingress apiVersion: extensions/v1beta1 metadata: name: kubernetes-dashboard namespace: kubernetes-dashboard annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/backend-protocol: HTTPS nginx.ingress.kubernetes.io/rewrite-target: / nginx.ingress.kubernetes.io/ssl-redirect: 'true' nginx.ingress.kubernetes.io/use-regex: 'true' spec: tls: - hosts: - dash.master.com secretName: kubernetes-dashboard-certs rules: - host: dash.master.com http: paths: - path: / pathType: ImplementationSpecific backend: serviceName: kubernetes-dashboard servicePort: 443 |
执行安装命令：
kubectl apply -f dash-ingress.yaml九，
获取token
kubectl describe sa default -n kube-system 输出如下：[root@localhost software]# kubectl describe sa default -n kube-systemName: defaultNamespace: kube-systemLabels: <none>Annotations: <none>Image pull secrets: <none>Mountable secrets: default-token-srkj8Tokens: default-token-srkj8Events: <none>kubectl describe secrets default-token-9hhsx -n kube-system输出如下：[root@localhost software]# kubectl describe secrets default-token-srkj8 -n kube-systemName: default-token-srkj8Namespace: kube-systemLabels: <none>Annotations: kubernetes.io/service-account.name: defaultkubernetes.io/service-account.uid: 34ed4707-ebe7-4699-85d1-09b20f3d0caeType: kubernetes.io/service-account-tokenData====token: eyJhbGciOiJSUzI1NiIsImtpZCI6IkdOMndKQ2FTUzd3c2ZhakVfSFFRekxFLXNQZGhUdUpVdGJyNFpsSTJmMkEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLXNya2o4Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIzNGVkNDcwNy1lYmU3LTQ2OTktODVkMS0wOWIyMGYzZDBjYWUiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGVmYXVsdCJ9.dGeBBZg-DzZJ7aUf-5FsNcm5x3JaGBMKMMaAa92-98PV7U-5pZQTcCvvw0Bi6nEGTFi8g_a6NQ3Tw43quPJV5FLMFgH9mQMnJXRtjjKomLjd4_GwYpK7cPaFuzwJWLqAXiddnEZmnyLj6D3qy5wc3QR5rgiQQ3QgrXKCZzXoYrlPg9dNUz3XqEgtxDlBYMFe43Gn9e8Xw7NOgydqKv0Qhxqjltx_nGJFw2fXIdoVBQQM1uC1BU37XqJJrh0wficXw57aB338W9ena38454V8pxWs2gYAlsOcCPJDAQb_tZA1e9JoHFWIwZ5VP_YHZC3MGTiVdjws6i8EpcPRM3QFkQca.crt: 1070 bytesnamespace: 11 byteskubectl describe secrets $(kubectl describe sa default -n kube-system | grep Mountable | awk 'NR == 2 {next} {print $3}') -n kube-system安装到这个阶段的时候，三个节点使用的镜像如下：
master节点：
[root@master opt]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
bitnami/kubectl 1.17.13-debian-10-r21 7022735edf5f 19 months ago 129MB
kubernetesui/metrics-scraper v1.0.6 48d79e554db6 20 months ago 34.5MB
quay.io/coreos/flannel v0.13.0 e708f4bb69e3 20 months ago 57.2MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy v1.19.3 cdef7632a242 20 months ago 118MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager v1.19.3 9b60aca1d818 20 months ago 111MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver v1.19.3 a301be0cd44b 20 months ago 119MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler v1.19.3 aaefbfa906bd 20 months ago 45.7MB
kubernetesui/dashboard v2.0.4 46d0a29c3f61 22 months ago 225MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd 3.4.13-0 0369cf4303ff 22 months ago 253MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns 1.7.0 bfe3a36ebd25 2 years ago 45.2MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause 3.2 80d28bedfe5d 2 years ago 683kB
registry.c7n.gzinfo/choerodon-tools/kubectl v1.15.2 2fad3003d792 2 years ago 52.5MB
slave1节点：
[root@slave1 ~]# docker images --digests
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller none <none> ae1739386d6a 7 months ago 285MB
registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller <none> sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a 435df390f367 17 months ago 279MB
jettech/kube-webhook-certgen v1.5.1 sha256:950833e19ade18cd389d647efb88992a7cc077abedef343fa59e012d376d79b7 a013daf8730d 19 months ago 44.7MB
kubernetesui/metrics-scraper v1.0.6 <none> 48d79e554db6 20 months ago 34.5MB
quay.io/coreos/flannel v0.13.0 <none> e708f4bb69e3 20 months ago 57.2MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy v1.19.3 <none> cdef7632a242 20 months ago 118MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler v1.19.3 <none> aaefbfa906bd 20 months ago 45.7MB
kubernetesui/dashboard v2.0.4 <none> 46d0a29c3f61 22 months ago 225MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd 3.4.13-0 <none> 0369cf4303ff 22 months ago 253MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns 1.7.0 <none> bfe3a36ebd25 2 years ago 45.2MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause 3.2 <none> 80d28bedfe5d 2 years ago 683kB
registry.cn-hangzhou.aliyuncs.com/google_containers/defaultbackend 1.4 <none> 846921f0feslave2节点：
[root@slave2 ~]# docker images --digests
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller v0.50.0 <none> ae1739386d6a 7 months ago 285MB
registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller <none> sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a 435df390f367 17 months ago 279MB
jettech/kube-webhook-certgen v1.5.1 <none> a013daf8730d 19 months ago 44.7MB
quay.io/coreos/flannel v0.13.0 <none> e708f4bb69e3 20 months ago 57.2MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy v1.19.3 <none> cdef7632a242 20 months ago 118MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler v1.19.3 <none> aaefbfa906bd 20 months ago 45.7MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns 1.7.0 <none> bfe3a36ebd25 2 years ago 45.2MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause 3.2 <none> 80d28bedfe5d 2 years ago 683kB
registry.cn-hangzhou.aliyuncs.com/google_containers/defaultbackend 1.4 <none> 846921f0fe0e 4 years ago 4.84MB
其中，kubernetesui/metrics-scraper这个镜像是dashboard信息收集插件，
kubernetesui/dashboard是主镜像，
registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller这个镜像的digest应该是3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a
十，nfs插件的安装
（1）
nfs服务的安装----三个节点都安装
yum install nfs nfs-utils rpcbind -y
systemctl enable nfs rpcbid && systemctl start nfs rpcbind（2）
nfs的配置文件编辑
[root@master ~]# cat /etc/exports
/data/k8s 10.244.0.0/16(rw,no_root_squash,no_subtree_check) 192.168.217.16(rw,no_root_squash,no_subtree_check) 192.168.217.0/24(rw,no_root_squash,no_subtree_check)（3）
建立存储点，给予存储点777权限--- 这一步是在master节点操作，别的节点不需要
mkdir -p /data/k8s
chmod -Rf 777 /data/k8s（4）
验证，在slave1或者2节点验证
systemctl restart nfs rpcbind
showmount -e master正确输出如下：
[root@master ~]# showmount -e master
Export list for master:
/data/k8s 192.168.217.0/24,10.244.0.0/16
（5）
使用helm安装（nfs-client-provisioner-0.1.1.tgz这个是helm的离线chart包）
helm install nfs-client-provisioner ./nfs-client-provisioner-0.1.1.tgz --set rbac.create=true --set persistence.enabled=true --set storageClass.name=nfs-provisioner --set persistence.nfsServer=192.168.217.16 --set persistence.nfsPath=/data/k8s --version 0.1.1 --namespace kube-system（6）设置默认storageclass
kubectl patch storageclass nfs-provisioner -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'十一，
nfs插件的验证性安装 redis集群
1，建立pvc
helm install redis ./redis-persistentvolumeclaim-0.1.0.tgz --set accessModes={ReadWriteOnce} --set requests.storage=256Mi --set storageClassName=nfs-provisioner --create-namespace --version 0.1.0 --namespace kube-system此时，可以查看一下pvc，pvc的名称叫redis，其实到这里的时候基本就已经表示该nfs插件是正常的了，因为pvc都是bound啦。
[root@master ~]# k get pvc -A
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
kube-system redis Bound pvc-751a32b6-8706-477b-8cad-d71e8e9f3ab8 256Mi RWO nfs-provisioner 26m
kube-system redis-data-redis-test-master-0 Bound pvc-f9193155-776c-42f4-a3f5-71e75f16416f 8Gi RWO nfs-provisioner 22m
kube-system redis-data-redis-test-replicas-0 Bound pvc-d5ea7d10-2ffa-402e-b3f1-8573a195ad6f 8Gi RWO nfs-provisioner 22m
kube-system redis-data-redis-test-replicas-1 Bound pvc-04203f8a-5907-48ce-9fc2-013e94313c3c 8Gi RWO nfs-provisioner 7m40s
kube-system redis-data-redis-test-replicas-2 Bound pvc-e1693689-b01b-4855-ab1c-b8f843be4e2e 8Gi RWO nfs-provisioner 6m41s
2，
安装redis
helm install redis-test ./redis-16.4.1.tgz --set persistence.enabled=true --set persistence.existingClaim=redis --set service.enabled=true --version 0.2.5 --namespace kube-system这个命令的输出如下：
NAME: redis-test
LAST DEPLOYED: Sat Jul 2 10:36:09 2022
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: redis
CHART VERSION: 16.4.1
APP VERSION: 6.2.6** Please be patient while the chart is being deployed **Redis™ can be accessed on the following DNS names from within your cluster:redis-test-master.kube-system.svc.cluster.local for read/write operations (port 6379)redis-test-replicas.kube-system.svc.cluster.local for read-only operations (port 6379)To get your password run:export REDIS_PASSWORD=$(kubectl get secret --namespace kube-system redis-test -o jsonpath="{.data.redis-password}" | base64 --decode)To connect to your Redis™ server:1. Run a Redis™ pod that you can use as a client:kubectl run --namespace kube-system redis-client --restart='Never' --env REDIS_PASSWORD=$REDIS_PASSWORD --image registry.hand-china.com/tools/redis:6.2.6-debian-10-r120 --command -- sleep infinityUse the following command to attach to the pod:kubectl exec --tty -i redis-client \--namespace kube-system -- bash2. Connect using the Redis™ CLI:REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h redis-test-masterREDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h redis-test-replicasTo connect to your database from outside the cluster execute the following commands:kubectl port-forward --namespace kube-system svc/redis-test-master : &REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h 127.0.0.1 -p
这里需要用的镜像是registry.hand-china.com_tools_redis_6.2.6-debian-10-r120.tar和redis4.0.11 这两个镜像，都被分配到slave1和slave2了。
linux下离线安装k8s集群1.19.4附带nfs存储（kubeadm方式）相关推荐
- linux下离线配置voltdb集群
linux下离线配置voltdb集群 系统环境:centos7.5最小化安装,因为是内网环境所以配置了私有yum源. 同一网络下的三台服务器,192.168.1.10.192.168.1.11.192 ...
- 从零开始离线安装k8s集群
本文主要用于在内网(离线)环境安装k8s集群:linux环境 centos7.6 主要步骤有: 安装docker 创建dokcer 私有镜像库 registry 安装kubernetes 安装flan ...
- 『Kubernetes』在Linux中快速安装K8S集群
- Centos离线安装Kubernetes集群
目录 kuberadmin离线安装K8s集群 环境准备 基础环境配置 安装docker 准备基础镜像以及安装包 镜像 集群初始化 设置.kube/config 安装网络组件 加入node节点 验证集群 ...
- 使用KubeKey安装K8S集群
KubeKey [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-9a5Ji2nE-1625797190511)(https://github.com/kubesphere ...
- breeze 可视安装 k8s 集群
breeze 可视安装 k8s 集群 官方文档: https://github.com/wise2c-devops/breeze/blob/master/BreezeManual-CN.md bree ...
- Linux下搭建Lotus Domino集群
Linux下搭建Lotus Domino 集群 本文内容是Linux平台下Lotus Domino服务器部署案例(http://chenguang.blog.51cto.com/350944/1334 ...
- Blazor+Dapr+K8s微服务之基于WSL安装K8s集群并部署微服务
前面文章已经演示过,将我们的示例微服务程序DaprTest1部署到k8s上并运行.当时用的k8s是Docker for desktop 自带的k8s,只要在Docker for desktop中启用 ...
- LINUX 下 配置MySQL数据库集群
LINUX 下 配置MySQL数据库集群 MySQL数据库集群进行正确配置步骤(1) 此文章主要向大家讲述的是对MySQL数据库集群进行正确配置的实际操作步骤,以及对其概念的讲述,如果你对其相关的实际 ...
最新文章
- 人工智能产业趋势和机遇！
- MyEclipse内安装与使用SVN
- PMcaff-干货| 内容营销可不仅仅是点击量而已
- Hi3516A开发--i2c tools安装和使用
- vueJs开发音乐播放器第二篇（点击歌单跳出详情页）
- 学习笔记02：直播串讲——3/22
- 数百万行自研代码都捐了，华为将欧拉捐赠给开放原子开源基金会
- （软件工程复习核心重点）第十二章软件项目管理-第一节：软件项目管理综述、估算软件规模和工作量估算
- 创建存储过程向表中循环加入数据
- mod_fcgid FcgidMaxRequestLen 131072 问题
- Inno Setup入门（三）——指定压缩方式
- LDR_DATA_TABLE_ENTRY结构得不到完整路径?
- android弹窗不能手动关闭_如何检测弹窗、并关闭相应的安卓弹窗
- 什么是函数指针 ？ 什么是指针函数？ int （*P）( ) 和int *p()有什么区别
- 微习惯养成，互联网产品成败的关键因素
- 使用DISM启用或禁用Windows功能
- markdown无法显示图片的问题
- Validform使用说明
- vulnhub之Hacker_Kid-v1.0.1
- 工作中使用了一些触发器