OpenStack虚拟机部署K8S集群，使用Cinder提供PV报错：mount: special device does not exist
2024-05-29 14:15:20
1、出错信息
kubelet挂载pv卷失败，从出错信息来看失败原因是设备不存在
# kubectl describe pods testpvcpod-cinder
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/7d0ca177-2624-4b4e-9353-222fb491f8cc /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
Output: Running scope as unit run-18372.scope.
mount: special device /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/7d0ca177-2624-4b4e-9353-222fb491f8cc does not existWarning FailedMount 6m6s kubelet, k8s-n1 Unable to attach or mount volumes: unmounted volumes=[testpv-cinder], unattached volumes=[default-token-lrww8 testpv-cinder]: timed out waiting for the conditionWarning FailedMount 116s (x4 over 6m1s) kubelet, k8s-n1 (combined from similar events): MountVolume.SetUp failed for volume "pvc-203a9760-85e9-4f58-91cd-081e9be96946" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/7d0ca177-2624-4b4e-9353-222fb491f8cc /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
Output: Running scope as unit run-21619.scope.
mount: special device /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/7d0ca177-2624-4b4e-9353-222fb491f8cc does not existWarning FailedMount 93s kubelet, k8s-n1 Unable to attach or mount volumes: unmounted volumes=[testpv-cinder], unattached volumes=[testpv-cinder default-token-lrww8]: timed out waiting for the condition
2、带Cinder PV pod部署流程
2.1、K8S集群信息
在OpenStack集群中部署了4个虚拟机，分别用于部署K8S Master和Node节点
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-m1 Ready master 112s v1.17.0
k8s-m2 Ready master 87s v1.17.0
k8s-m3 Ready master 21s v1.17.0
k8s-n1 Ready <none> 13s v1.17.0
2.2、创建Cinder Storage Class
# cat cinder-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: standard
provisioner: kubernetes.io/cinder
reclaimPolicy: Delete
volumeBindingMode: Immediate
# kubectl apply -f cinder-sc.yaml
storageclass.storage.k8s.io/standard created
# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
standard kubernetes.io/cinder Delete Immediate false 5s
2.3、创建带动态Cinder PV的POD
# cat busybox-cinder-dym.yaml
apiVersion: v1
kind: Pod
metadata:name: testpvcpod-cinder
spec:containers:- name: busybox-pvc-cinderimage: busyboxcommand:- sh- -c- "while true; do date > /home/loop; sleep 10m; done"imagePullPolicy: IfNotPresentvolumeMounts:- name: testpv-cindermountPath: /homevolumes:- name: testpv-cinderpersistentVolumeClaim:claimName: cinder-pvc---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: cinder-pvc
spec:accessModes:- ReadWriteOncestorageClassName: standardresources:requests:storage: 1Gi
# kubectl apply -f busybox-cinder-dym.yaml
pod/testpvcpod-cinder created
persistentvolumeclaim/cinder-pvc created
# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
nfs-pv 1Gi RWX Recycle Bound default/nfs-pvc slow 26h
pvc-ce961d70-ea7f-4f23-9cf7-7de782951136 1Gi RWO Delete Bound default/cinder-pvc standard 7m46s
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cinder-pvc Bound pvc-ce961d70-ea7f-4f23-9cf7-7de782951136 1Gi RWO standard 7m49s
nfs-pvc Bound nfs-pv 1Gi RWX slow 26h
PV、PVC和POD都已经创建好，但是POD处于ContainerCreating状态
# kubectl get pods
NAME READY STATUS RESTARTS AGE
testpvcpod 1/1 Running 0 25h
testpvcpod-cinder 0/1 ContainerCreating 0 7m45s
没有running的原因就是因为本文开头的问题
3、解决问题
3.1、查看Cinder PV卷是否挂载到节点上
k8s-n1节点上的/dev/vdb就是这个PV
[root@k8s-n1 ~]# fdisk -l
Disk /dev/vda: 54.8 GB, 54760833024 bytes, 106954752 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00072170Device Boot Start End Blocks Id System
/dev/vda1 * 2048 2099199 1048576 83 Linux
/dev/vda2 2099200 104857599 51379200 8e Linux LVMDisk /dev/mapper/centos-root: 47.2 GB, 47240445952 bytes, 92266496 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/mapper/centos-swap: 5368 MB, 5368709120 bytes, 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/vdb: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
3.2、PVC对应的目录不存在
[root@k8s-n1 ~]# ls -l /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
ls: cannot access /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946: No such file or directory
3.3、手工创建目录
[root@k8s-n1 ~]# mkdir -p /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
[root@k8s-n1 ~]# ls -l /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
total 0
3.4、手工挂载卷
[root@k8s-n1 ~]# mkfs.xfs /dev/vdb
meta-data=/dev/vdb isize=512 agcount=4, agsize=65536 blks= sectsz=512 attr=2, projid32bit=1= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=262144, imaxpct=25= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@k8s-n1 ~]# mount /dev/vdb /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
[root@k8s-n1 ~]# df
/dev/vdb 1038336 32992 1005344 4% /var/lib/kubelet/pods/71b27d9f-3f99-4b1a-a3b5-527ebeea409e/volumes/kubernetes.io~cinder/pvc-203a9760-85e9-4f58-91cd-081e9be96946
3.5、POD处于running状态
# kubectl get pods
NAME READY STATUS RESTARTS AGE
testpvcpod 1/1 Running 0 42h
testpvcpod-cinder 1/1 Running 0 9m54s
查看卷已经挂载到pod上
# kubectl exec -it testpvcpod-cinder df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 46110724 5801528 40309196 13% /
tmpfs 65536 0 65536 0% /dev
tmpfs 4004356 0 4004356 0% /sys/fs/cgroup
/dev/vdb 1038336 32996 1005340 3% /home
4、问题分析
因为本环境用的是K8S原生的Cinder驱动，没有使用CSI，而原生的Cinder驱动在1.17.0版本及后续版本将逐渐废弃，转为CSI。所以这块代码可能存在bug。即k8s master虽然调用OpenStack接口为pod创建并挂载了卷，但是kubelet不知道卷信息，无法做自动mount操作。
后续有空分析下这块代码，并验证下CSI Cinder的使用
OpenStack虚拟机部署K8S集群，使用Cinder提供PV报错：mount: special device does not exist相关推荐
- 最简单部署k8s集群（一）
之前的电脑配置不太够,学习K8s就使用"minikube"本机单机部署k8s环境,下载minikube二进制文件启动就可以.现在电脑配置好一些,使用虚拟机部署k8s集群. 部署k8 ...
- kubeadm部署k8s集群
1.准备环境 虚拟机操作系统: Centos7 角色 IP Master 192.168.150.140 Node1 192.168.150. ...
- linux上部署K8S集群
部署K8S集群 服务器硬件要求:三台虚拟机服务器,操作系统都为centos: 硬盘最低配置:内存2GB,CPU2核,硬盘30GB. 准备环境 master 192.168.200.110 node ...
- kubekey部署k8s集群
基本介绍:KubeKey 是 KubeSphere 社区开源的一款高效集群部署工具,运行时默认使用 Docker , 也可对接 Containerd CRI-O iSula 等 CRI 运行时,且 E ...
- RKE安装部署K8S集群、Rancher
服务器准备:三台虚拟机(master:1,node:2:这里选用的阿里云ECS) OS hostname 内网IP Centos7 joker-master-1 172.27.31.149 Cento ...
- k8s入门：裸机部署 k8s 集群
系列文章 第一章:✨ k8s入门:裸机部署 k8s 集群 第二章:✨ k8s入门:部署应用到 k8s 集群 第三章:✨ k8s入门:service 简单使用 第四章:✨ k8s入门:StatefulS ...
- 【02】Kubernets：使用 kubeadm 部署 K8S 集群
写在前面的话 通过上一节,知道了 K8S 有 Master / Node 组成,但是具体怎么个组成法,就是这一节具体谈的内容.概念性的东西我们会尽量以实验的形式将其复现. 部署 K8S 集群 互联网常 ...
- kubeadm部署K8S集群并使用containerd做容器运行时
kubeadm部署K8S集群并使用containerd做容器运行时(内容全部实战验证有任何问题欢迎留言咨询讨论) 前言 去年12月份,当Kubernetes社区宣布1.20版本之后会逐步弃用docke ...
- 如何通过rancher部署k8s集群
目录 1 安装前准备 2 安装rancher 3 配置Rancher 4 创建k8s集群 最近的工作中需要使用到K8S,而面临的第一个问题就是如何部署一个K8S集群环境.现有多种部署方式,如:kube ...
最新文章
- Web用户控件的一个毛病
- select下拉框下拉跳转代码
- python程序员需要掌握哪些技术-高级的python 后端程序员， 应该掌握哪些技能点 ？...
- linux获取ip地址的接口,在Linux上获取接口的IP地址
- 你是如何坚持读完《算法导论》这本书的？(帖子收集)
- mysql 定义XML字段_MyBatis之基于XML的属性与列名映射
- MongoDB投影字段
- 1-7：学习shell之透过shell看世界
- 文件隐藏服务器版本信息吗,隐藏/屏蔽服务器信息与web软件版本信息
- eplan如何导入access_EPLAN脚本从入门到应用-2
- HDU 6321（状压dp）
- 计算机键盘的认识 功能,笔记本电脑键盘的功能_笔记本电脑键盘的认识
- 阿里天池大数据竞赛第一名，如何用AI检测肺癌
- 【CF833D】Red-Black Cobweb（点分治）
- 【历史上的今天】9 月 6 日：Facebook 收购 Instagram；华为发布 5G 芯片；惠普并购康柏
- iOS：quartz2D绘图（绘制渐变图形）
- c语言出现开头结尾多空格,C语言除去字符串开头和末尾空格之trim()函数的实现...
- lua attempt to index field ? a nil
- 易飞9.08-9.0.12安装及算号
- cypress学习笔记二：目录结构及登录测试用例