k8s中使用基于nfs的storageclass

Gitlab地址：https://github.com/kubernetes-retired/external-storage/tree/master/nfs-client

1．搭建nfs服务

(1)nfs服务端的安装配置

# 安装nfs
~]# yum -y install nfs-utils rpcbind
# 创建nfs共享目录及设置权限
~]# mkdir /data/k8sdata -p
~]# chmod 755 /data/k8sdata -R
# 配置nfs
~]# cat /etc/exports
/data/k8sdata *(rw,sync,no_root_squash)

nfs主要配置说明：

* ：表示任何⼈都有权限连接，这儿也可以配置为⼀个⽹段，⼀个 IP，或者域名

rw：权限为读写

sync：表示⽂件同时写⼊硬盘和内存

no_root_squash：当登录 NFS 主机使⽤共享⽬录的使⽤者是 root 时，其权限将被转换成为匿名使⽤者，通常它的 UID 与 GID，都会变成 nobody 身份

# 启动nfs，需要先启动rpcbind再启动nfs
~]# systemctl start rpcbind
~]# systemctl enable rpcbind
~]# systemctl status rpcbind
● rpcbind.service - RPC bind serviceLoaded: loaded (/usr/lib/systemd/system/rpcbind.service; enabled; vendor preset: enabled)Active: active (running) since Sun 2020-08-09 14:53:03 CST; 50s agoMain PID: 15672 (rpcbind)CGroup: /system.slice/rpcbind.service└─15672 /sbin/rpcbind -w
~]# systemctl start nfs
~]# systemctl enable nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
~]# systemctl status nfs
● nfs-server.service - NFS server and servicesLoaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)Drop-In: /run/systemd/generator/nfs-server.service.d└─order-with-mounts.confActive: active (exited) since Sun 2020-08-09 14:57:22 CST; 17s agoMain PID: 16369 (code=exited, status=0/SUCCESS)CGroup: /system.slice/nfs-server.service
# 查看具体的挂载权限
~]# cat /var/lib/nfs/etab
/data/k8sdata *(rw,sync,wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,no_pnfs,anonuid=65534,anongid=65534,sec=sys,rw,secure,no_root_squash,no_all_squash)

(2)nfs客户端的安装配置

Nfs客户端需要安装在使用nfs的各个节点上，即k8s集群的各个node节点上。

# 在客户端上安装启动nfs
~]# yum -y install nfs-utils rpcbind
~]# systemctl start rpcbind
~]# systemctl enable rpcbind
~]# systemctl start nfs
~]# systemctl enable nfs
# 查看nfs共享
~]# showmount -e 192.168.16.133
Export list for 192.168.16.133:
/data/k8sdata *

2、创建基于nfs的storageclass

(1)storageclass

要使⽤StorageClass，我们就得安装对应的⾃动配置程序，⽐如我们这⾥存储后端使⽤的是 nfs，那么我们就需要使⽤到⼀个 nfs-client的⾃动配置程序，我们也叫它 Provisioner，这个程序帮我们在配置好的nfs 服务器上⾃动创建持久卷，也就是⾃动帮我们创建PV。⾃动创建的PV以${namespace}-${pvcName}-${pvName}这样的命名格式创建在NFS服务器上的共享数据⽬录中，⽽当这个PV被回收后会以archieved-${namespace}-${pvcName}-${pvName}这样的命名格式存在NFS 服务器上。

(2)、安装配置nfs storageclass

# 拉取github相关的资源至本地
~]# git clone https://github.com/kubernetes-retired/external-storage.git
~]# cd external-storage/tree/master/nfs-client/deploy/
# 将deployment.yaml配置文件中的nfs服务器和路径修改为自己nfs服务器和路径
]# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: nfs-client-provisionerlabels:app: nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default
spec:replicas: 1strategy:type: Recreateselector:matchLabels:app: nfs-client-provisionertemplate:metadata:labels:app: nfs-client-provisionerspec:serviceAccountName: nfs-client-provisionercontainers:- name: nfs-client-provisionerimage: quay.io/external_storage/nfs-client-provisioner:latestvolumeMounts:- name: nfs-client-rootmountPath: /persistentvolumesenv:- name: PROVISIONER_NAMEvalue: fuseim.pri/ifs- name: NFS_SERVERvalue: 192.168.16.133- name: NFS_PATHvalue: /data/k8sdatavolumes:- name: nfs-client-rootnfs:server: 192.168.16.133path: /data/k8sdata
# 修改完成后创建该deployment控制器
deploy]# kubectl apply -f deployment.yaml
deployment.apps/nfs-client-provisioner created
# 创建sa，并绑定相应的权限
deploy]# kubectl apply -f rbac.yaml
serviceaccount/nfs-client-provisioner created
# 创建storageclass
deploy]# kubectl apply -f class.yaml
storageclass.storage.k8s.io/managed-nfs-storage created
# 查看创建的资源
deploy]# kubectl get pods
NAME                                      READY   STATUS             RESTARTS   AGE
nfs-client-provisioner-7d4d65c7b9-l487r   1/1     Running            0          8s
deploy]# kubectl get storageclass
NAME                  PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
managed-nfs-storage   fuseim.pri/ifs   Delete          Immediate           false                  22h

3、使用基于nfs的storageclass

（1）创建pvc调用storageclass动态提供pv

创建了storageclass后，创建pvc之前不需要创建相关的pv,只需调用storageclass提供pv的动态供给。

# 要创建的pvc内容如下
deploy]# cat test-claim.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:name: test-claimannotations:volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:accessModes:- ReadWriteManyresources:requests:storage: 1Mi
# 创建pvc
deploy]# kubectl apply -f test-claim.yaml
persistentvolumeclaim/test-claim created
deploy]# kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
test-claim   Bound    pvc-15531f5b-b439-43cc-82a8-7152a5ffd50f   1Mi        RWX            managed-nfs-storage   5s

（2）在pod控制器中使用storageclass

在实际⼯作中，使⽤ StorageClass 更多的是StatefulSet控制器管理的pod，StatefulSet控制器中我们也可以通过⼀个volumeClaimTemplates属性直接使⽤ StorageClass。实际上volumeClaimTemplates下⾯就是⼀个PVC对象的模板，类似于StatefulSet下⾯的template，实际上就是⼀个 Pod 的模板，我们用这种模板就可以动态的去创建pvc对象了。

# statefulset控制器内容
]# cat statefulset-nfs-storageclass.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:name: nfs-test
spec:serviceName: "nginx"replicas: 2selector:matchLabels:app: nfs-testtemplate:metadata:labels:app: nfs-testspec:terminationGracePeriodSeconds: 5containers:- name: nginximage: nginx:1.18.0ports:- name: httpcontainerPort: 80volumeMounts:- name: webmountPath: /usr/share/nginx/htmlvolumeClaimTemplates:- metadata:name: webannotations:volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"spec:accessModes: ["ReadWriteOnce"]resources:requests:storage: 1Gi
# 创建
]# kubectl apply -f statefulset-nfs-storageclass.yaml
# 查看创建的pvc
]# kubectl get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
web-nfs-test-0   Bound    pvc-f956a6c8-5382-4a0a-a7ed-d19ac07d83c2   1Gi        RWO            managed-nfs-storage   2m2s
web-nfs-test-1   Bound    pvc-2a25a995-37db-4173-be1e-7a94ea4f9e8f   1Gi        RWO            managed-nfs-storage   119s
# 创建pvc后会在nfs上生成相关的文件
~]# ll /data/k8sdata/
total 0
drwxrwxrwx 2 root root 6 Aug 11 22:06 default-test-claim-pvc-15531f5b-b439-43cc-82a8-7152a5ffd50f
drwxrwxrwx 2 root root 6 Aug 11 22:59 default-web-nfs-test-0-pvc-f956a6c8-5382-4a0a-a7ed-d19ac07d83c2
drwxrwxrwx 2 root root 6 Aug 11 22:59 default-web-nfs-test-1-pvc-2a25a995-37db-4173-be1e-7a94ea4f9e8f