pod一直处于ContainerCreating，查看报错信息为挂载错误MountVolume.SetUp failed for volume

背景，在搭建redis集群时，使用的是nfs挂载卷，中途我好像把挂载盘的文件移走了，当我再次启动pod时就出现挂载错误。

[root@master redis-cluster-sts]# kubectl apply -f redis-sts.yml
configmap/redis-cluster created
statefulset.apps/redis-cluster created
[root@master redis-cluster-sts]# kubectl get pods
NAME                                      READY   STATUS              RESTARTS   AGE
nfs-client-provisioner-78cbf94495-4zwqj   1/1     Running             4          39h
redis-cluster-0                           0/1     ContainerCreating   0          4s
[root@master redis-cluster-sts]# kubectl describe pods redis-cluster-0
Name:           redis-cluster-0
Namespace:      default
Priority:       0
Node:           node3/172.31.17.120
Start Time:     Fri, 15 May 2020 10:03:14 +0800
Labels:         app=redisappCluster=redis-clustercontroller-revision-hash=redis-cluster-75f6c9b7c8statefulset.kubernetes.io/pod-name=redis-cluster-0
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  StatefulSet/redis-cluster
Containers:redis:Container ID:Image:         redis:latestImage ID:Ports:         6379/TCP, 16379/TCPHost Ports:    0/TCP, 0/TCPCommand:/conf/update-node.shredis-server/conf/redis.confState:          WaitingReason:       ContainerCreatingReady:          FalseRestart Count:  0Environment:POD_IP:          (v1:status.podIP)METADATA_NAME:  redis-cluster-0 (v1:metadata.name)Mounts:/conf from conf (rw)/data from data (rw)/var/run/secrets/kubernetes.io/serviceaccount from default-token-jpxkg (ro)
Conditions:Type              StatusInitialized       TrueReady             FalseContainersReady   FalsePodScheduled      True
Volumes:data:Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)ClaimName:  data-redis-cluster-0ReadOnly:   falseconf:Type:      ConfigMap (a volume populated by a ConfigMap)Name:      redis-clusterOptional:  falsedefault-token-jpxkg:Type:        Secret (a volume populated by a Secret)SecretName:  default-token-jpxkgOptional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300snode.kubernetes.io/unreachable:NoExecute for 300s
Events:Type     Reason       Age        From               Message----     ------       ----       ----               -------Normal   Scheduled    <unknown>  default-scheduler  Successfully assigned default/redis-cluster-0 to node3Warning  FailedMount  16s        kubelet, node3     MountVolume.SetUp failed for volume "pvc-6cea9fce-006d-40f9-8e4a-3c27f323ef80" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/d73302e3-da43-4d4f-b218-68e0a6cbbcb6/volumes/kubernetes.io~nfs/pvc-6cea9fce-006d-40f9-8e4a-3c27f323ef80 --scope -- mount -t nfs vol:/share/nfs/default-data-redis-cluster-0-pvc-6cea9fce-006d-40f9-8e4a-3c27f323ef80 /var/lib/kubelet/pods/d73302e3-da43-4d4f-b218-68e0a6cbbcb6/volumes/kubernetes.io~nfs/pvc-6cea9fce-006d-40f9-8e4a-3c27f323ef80
Output: Running scope as unit run-18199.scope.
mount.nfs: mounting vol:/share/nfs/default-data-redis-cluster-0-pvc-6cea9fce-006d-40f9-8e4a-3c27f323ef80 failed, reason given by server: No such file or directoryWarning  FailedMount  15s  kubelet, node3  MountVolume.SetUp failed for volume "pvc-6cea9fce-006d-40f9-8e4a-3c27f323ef80" : mount failed: exit status 32

排错：
1、以为是nfs出问题了，于是在外部新建一个目录，挂载到vol的/share/nfs，发现是挂载成功的。

#查看vol分享出来的目录
[root@master redis-cluster-sts]# showmount -e vol
Export list for vol:
/share                    172.31.0.0/16
/data/opv                 172.31.0.0/16
/home/ttebdadmin/nfs/data 172.31.0.0/16
#创建一个文件
[root@master redis-cluster-sts]# mkdir share
#将vol:/share/nfs挂载至此目录下
[root@master redis-cluster-sts]# mount vol:/share/nfs share/
#挂载成功
[root@master redis-cluster-sts]# mount | grep vol:/share/nfs
vol:/share/nfs on /usr/local/ttebd/k8s-base-v16.2/redis-cluster-sts/share type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=172.31.17.51,local_lock=none,addr=172.31.17.54)
#查看nfs目录下的文件已经能够查看了
[root@master redis-cluster-sts]# ls share/
default-redis-cluster-data-redis-app-0-pvc-e2069f3b-d370-4ad1-aeca-ac64f8c426ad
kafka-datadir-zk-0-pvc-f4dd3352-f8ba-4977-b154-882ea509e7af
kafka-datadir-zk-1-pvc-701b3432-aa3b-440e-8cac-81b1e6de69a3
kafka-datadir-zk-2-pvc-8bbd84fb-fdfc-4aae-ab98-29fa971d19b9
nodes.conf
update-node.sh

至此说明nfs是没问题的，上面报错表示文件路径找不到，再看下报错信息vol:/share/nfs/default-data-redis-cluster-0-pvc-6cea9fce-006d-40f9-8e4a-3c27f323ef80 是default-data-XXX文件找不到，查看nfs目录下确实是没有这个文件的，于是我想到是不是有缓存，于是换了个挂载路径，重启了主机，发现还是不行，一直会找这个地址，最终发现其实是pvc，查看pvc和pv

[root@master redis-cluster-sts]# kubectl get pvc
NAME                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-redis-cluster-0             Bound    pvc-6cea9fce-006d-40f9-8e4a-3c27f323ef80   1Gi        RWO            nfs-storage    36h
data-redis-cluster-1             Bound    pvc-527fe5c7-0cb4-4816-825d-44e471299206   1Gi        RWO            nfs-storage    36h
data-redis-cluster-2             Bound    pvc-bad20305-2b9c-4c81-8383-5b2a011c18c1   1Gi        RWO            nfs-storage    36h
data-redis-cluster-3             Bound    pvc-40f40264-0457-4b50-8f80-fa57571d0acf   1Gi        RWO            nfs-storage    36h
data-redis-cluster-4             Bound    pvc-eafb5c30-b432-4d8a-9cb4-9b52de34b0d4   1Gi        RWO            nfs-storage    36h
data-redis-cluster-5             Bound    pvc-659e52ec-bf6e-4d51-b549-91dfbf0a941a   1Gi        RWO            nfs-storage    36h
redis-cluster-data-redis-app-0   Bound    pvc-e2069f3b-d370-4ad1-aeca-ac64f8c426ad   1G         RWO,RWX        nfs-storage    36h
[root@master redis-cluster-sts]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                    STORAGECLASS   REASON   AGE
pvc-40f40264-0457-4b50-8f80-fa57571d0acf   1Gi        RWO            Retain           Bound    default/data-redis-cluster-3             nfs-storage             36h
pvc-527fe5c7-0cb4-4816-825d-44e471299206   1Gi        RWO            Retain           Bound    default/data-redis-cluster-1             nfs-storage             36h
pvc-659e52ec-bf6e-4d51-b549-91dfbf0a941a   1Gi        RWO            Retain           Bound    default/data-redis-cluster-5             nfs-storage             36h
pvc-6cea9fce-006d-40f9-8e4a-3c27f323ef80   1Gi        RWO            Retain           Bound    default/data-redis-cluster-0             nfs-storage             36h
pvc-701b3432-aa3b-440e-8cac-81b1e6de69a3   1G         RWX            Retain           Bound    kafka/datadir-zk-1                       nfs-storage             35h
pvc-8bbd84fb-fdfc-4aae-ab98-29fa971d19b9   1G         RWX            Retain           Bound    kafka/datadir-zk-2                       nfs-storage             35h
pvc-bad20305-2b9c-4c81-8383-5b2a011c18c1   1Gi        RWO            Retain           Bound    default/data-redis-cluster-2             nfs-storage             36h
pvc-e2069f3b-d370-4ad1-aeca-ac64f8c426ad   1G         RWO,RWX        Retain           Bound    default/redis-cluster-data-redis-app-0   nfs-storage             36h
pvc-eafb5c30-b432-4d8a-9cb4-9b52de34b0d4   1Gi        RWO            Retain           Bound    default/data-redis-cluster-4             nfs-storage             36h
pvc-f4dd3352-f8ba-4977-b154-882ea509e7af   1G         RWX            Retain           Bound    kafka/datadir-zk-0                       nfs-storage             35h

再仔细查看上面的报错

  Warning  FailedMount  16s        kubelet, node3     MountVolume.SetUp failed for volume "pvc-6cea9fce-006d-40f9-8e4a-3c27f323ef80" : mount failed: exit status 32

是挂载pv时报错，查看pv信息

[root@master redis-cluster-sts]# kubectl describe pv pvc-40f40264-0457-4b50-8f80-fa57571d0acf
Name:            pvc-40f40264-0457-4b50-8f80-fa57571d0acf
Labels:          <none>
Annotations:     pv.kubernetes.io/provisioned-by: fuseim.pri/ifs
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    nfs-storage
Status:          Bound
Claim:           default/data-redis-cluster-3
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        1Gi
Node Affinity:   <none>
Message:
Source:Type:      NFS (an NFS mount that lasts the lifetime of a pod)Server:    volPath:      /share/nfs/default-data-redis-cluster-3-pvc-40f40264-0457-4b50-8f80-fa57571d0acfReadOnly:  false
Events:        <none>

pod启动时查找到对应的pv，然后查找pv对应的路径，发现文件路径不存在，于是报错，解决办法
1、新建相关的文件，例如在vol的/share/nfs目录下新建default-data-redis-cluster-0-pvc-6cea9fce-006d-40f9-8e4a-3c27f323ef80文件。

#vol上操作，新建文件
[root@vol nfs]# mkdir default-data-redis-cluster-0-pvc-6cea9fce-006d-40f9-8e4a-3c27f323ef80
#赋予权限
[root@vol nfs]# chmod 777 default-data-redis-cluster-0-pvc-6cea9fce-006d-40f9-8e4a-3c27f323ef80
[root@vol nfs]# ls
default-data-redis-cluster-0-pvc-6cea9fce-006d-40f9-8e4a-3c27f323ef80
default-redis-cluster-data-redis-app-0-pvc-e2069f3b-d370-4ad1-aeca-ac64f8c426ad
kafka-datadir-zk-0-pvc-f4dd3352-f8ba-4977-b154-882ea509e7af
kafka-datadir-zk-1-pvc-701b3432-aa3b-440e-8cac-81b1e6de69a3
kafka-datadir-zk-2-pvc-8bbd84fb-fdfc-4aae-ab98-29fa971d19b9
nodes.conf
update-node.sh# master上查看pod状态，发现redis-cluster-0这个pod跑起来了，要想其他的也跑起来，需要把其他的文件也补起来
[root@master redis-cluster-sts]# kubectl get pods
NAME                                      READY   STATUS              RESTARTS   AGE
nfs-client-provisioner-78cbf94495-4zwqj   1/1     Running             4          40h
redis-cluster-0                           1/1     Running             0          44m
redis-cluster-1                           0/1     ContainerCreating   0          15m

上面那种办法在找不到之前的文件的情况下恢复起来很麻烦，最简单的办法就是删除pvc和pv，让它自己重新创建文件

2、删除pvc及pv,重新启动pod即可。

[root@master redis-cluster-sts]# kubectl delete -f redis-sts.yml
configmap "redis-cluster" deleted
statefulset.apps "redis-cluster" deleted#删除pvc，注意0-5都需删除
[root@master redis-cluster-sts]# kubectl delete pvc data-redis-cluster-0#若pvc一直处于Terminating状态，则可使用patch直接删除
[root@master redis-cluster-sts]# kubectl patch pvc data-redis-cluster-0  -p '{"metadata":{"finalizers":null}}' -n default
persistentvolumeclaim/data-redis-cluster-0 patched#查看pv
[root@master redis-cluster-sts]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                                    STORAGECLASS   REASON   AGE
pvc-40f40264-0457-4b50-8f80-fa57571d0acf   1Gi        RWO            Retain           Released   default/data-redis-cluster-3             nfs-storage             36h
pvc-527fe5c7-0cb4-4816-825d-44e471299206   1Gi        RWO            Retain           Released   default/data-redis-cluster-1             nfs-storage             36h
pvc-659e52ec-bf6e-4d51-b549-91dfbf0a941a   1Gi        RWO            Retain           Released   default/data-redis-cluster-5             nfs-storage             36h
pvc-6cea9fce-006d-40f9-8e4a-3c27f323ef80   1Gi        RWO            Retain           Released   default/data-redis-cluster-0             nfs-storage             36h
pvc-bad20305-2b9c-4c81-8383-5b2a011c18c1   1Gi        RWO            Retain           Released   default/data-redis-cluster-2             nfs-storage             36h
pvc-eafb5c30-b432-4d8a-9cb4-9b52de34b0d4   1Gi        RWO            Retain           Released   default/data-redis-cluster-4             nfs-storage             36h#删除pv，注意0-5都需删除
[root@master redis-cluster-sts]# kubectl delete pv pvc-527fe5c7-0cb4-4816-825d-44e471299206
persistentvolume "pvc-527fe5c7-0cb4-4816-825d-44e471299206" deleted

重新运行redis-sts.yaml,查看pod运行正常

[root@master redis-cluster-sts]# kubectl apply -f redis-sts.yml
configmap/redis-cluster created
statefulset.apps/redis-cluster created
[root@master redis-cluster-sts]# kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-78cbf94495-4zwqj   1/1     Running   4          40h
redis-cluster-0                           1/1     Running   0          70s
redis-cluster-1                           1/1     Running   0          66s
redis-cluster-2                           1/1     Running   0          63s
redis-cluster-3                           1/1     Running   0          59s
redis-cluster-4                           1/1     Running   0          55s
redis-cluster-5                           1/1     Running   0          52s

pod一直处于ContainerCreating，查看报错信息为挂载错误MountVolume.SetUp failed for volume相关推荐

解决argo workflow报错：MountVolume.SetUp failed for volume “docker-sock“ : hostPath type check failed
提交workflow时报错: MountVolume.SetUp failed for volume "docker-sock" : hostPath type check fai ...
戴尔服务器硬件报错信息LCD液晶面板错误信息
戴尔服务器13代硬件报错信息代码大全发布 E1114 Temp Ambient 系统周围环境温度超出允许范围. E1116 Temp Memory 内存已超过允许温度,系统已将其禁用以防止损坏组件. ...
戴尔服务器系统板电压超出范围器,戴尔服务器硬件报错信息LCD液晶面板错误信息...
戴尔服务器13代硬件报错信息代码大全发布 E1114 Temp Ambient 系统周围环境温度超出允许范围. E1116 Temp Memory 内存已超过允许温度,系统已将其禁用以防止损坏组件. ...
php js 报错信息,JavaScript中错误异常的分析（附示例）
本篇文章给大家带来的内容是关于JavaScript中错误异常的分析(附示例),有一定的参考价值,有需要的朋友可以参考一下,希望对你有所帮助. 我的建议是不要隐藏错误,勇敢地抛出来.没有人会因为代码出现 ...
mysql报错信息1067_mysql 1067错误报错！这个是mysqld - -console得到的信息
该楼层疑似违规已被系统折叠隐藏此楼查看此楼 2017-09-20 17:06:11 0 [Warning] gethostname failed, using 'localhost' as host ...
05-【公司解决问题】公司解决问题5之k8s问题(部署单元挂载存储卷启动报错：nfs挂载错误wrong fs type, bad option, bad superblock)
一.报错原因: mount: wrong fs type, bad option, bad superblock on 10.202.35.160:/CHENYI_TEST_SIT, m ...
android查看报错日志,android运行错误日志帮看下不懂啊
该楼层疑似违规已被系统折叠隐藏此楼查看此楼 xmlns:tools="http://schemas.android.com/tools" android:id="@id ...
web应用F12查看报错（前后端bug判断、2XX/3XX/4XX/5XX常见状态码解析）
chrom浏览器为例 (1)打开开发者工具,在浏览器菜单栏选择工具-开发者工具,快捷键是F12 (2)打开之后切换到Network页签,操作就可以看到请求响应 (3)再选择响应的链接,切换到Previ ...
【elasticsearch系列】双击elasticsearch.bat闪退，日志排查报错信息
打开my-application.log日志文件,寻找报错信息,如下: [1] bootstrap checks failed [1]: the default discovery settings ...

pod一直处于ContainerCreating，查看报错信息为挂载错误MountVolume.SetUp failed for volume

pod一直处于ContainerCreating，查看报错信息为挂载错误MountVolume.SetUp failed for volume相关推荐

最新文章

热门文章