rook 排错记录 + Orphaned pod found kube-controller-manager的日志输出

1.查看rook-agent(重要)和mysql-wordpress 的日志，如下：

MountVolume.SetUp failed for volume "pvc-f002e1fe-469c-11e8-9dca-90b8d0599f2f" : mount command failed, status: Failure, reason: Rook: Error getting RPC client: error connecting to socket /usr/libexec/kubernetes/kubelet-plugins/volume/exec/rook.io~rook/.rook.sock: dial unix /usr/libexec/kubernetes/kubelet-plugins/volume/exec/rook.io~rook/.rook.sock: connect: no such file or directory

解决：

rook-operator.yml

env:- name: FLEXVOLUME_DIR_PATH  value: "/var/lib/kubelet/volumeplugins"

kubelet 启动加上

--volume-plugin-dir=/var/lib/kubelet/volumeplugins

2. kubectl describe pod wordpress-69895668dd-zfkvq

Unable to mount volumes for pod "wordpress-mysql-5bbddff49c-vbv4q_default(f2949440-e838-11e8-999c-06c9540001ea)": timeout expired waiting for volumes to attach or mount for pod "default"/"wordpress-mysql-5bbddff49c-vbv4q". list of unmounted volumes=[mysql-persistent-storage]. list of unattached volumes=[mysql-persistent-storage default-token-sp8rj]
Warning FailedMount 58s (x2 over 1m) kubelet, node3 MountVolume.SetUp failed for volume "pvc-b044b1d8-e819-11e8-84c3-060f100001e9" : mount command failed, status: Failure, reason: Rook: Mount volume failed: failed to attach volume replicapool/pvc-b044b1d8-e819-11e8-84c3-060f100001e9: failed to map image replicapool/pvc-b044b1d8-e819-11e8-84c3-060f100001e9 cluster rook-ceph. failed to map image replicapool/pvc-b044b1d8-e819-11e8-84c3-060f100001e9: Failed to complete 'rbd': signal: interrupt. . output:

方法：

进入容器手动执行

重启kubelet systemctl restart kubelet -- 不起作用

直接重启所有host机器，删除wordpress 和 mysql pod ,机器重启后删除有错误osd pod ,等10分钟后

default wordpress-69895668dd-zfkvq 1/1 Running 0 14h 10.233.71.43 node3
default wordpress-mysql-5bbddff49c-vbv4q 1/1 Running 0 14h 10.233.71.42 node3

###
1. 先把镜像都下载下来
2. 把pod个数设置为0
3. 把pv , pvc都删除
4.重新apply就可以

1.MountVolume.SetUp failed for volume "pvc-f002e1fe-469c-11e8-9dca-90b8d0599f2f" : mount command failed, status: Failure, reason: Rook: Error getting RPC client: error connecting to socket /usr/libexec/kubernetes/kubelet-plugins/volume/exec/rook.io~rook/.rook.sock: dial unix /usr/libexec/kubernetes/kubelet-plugins/volume/exec/rook.io~rook/.rook.sock: connect: no such file or directory

解决：

rook-operator.yml

env:- name: FLEXVOLUME_DIR_PATH  value: "/var/lib/kubelet/volumeplugins"

kubelet 启动加上

--volume-plugin-dir=/var/lib/kubelet/volumeplugins

转载于:https://www.cnblogs.com/hixiaowei/p/9962018.html

rook 排错记录 + Orphaned pod found kube-controller-manager的日志输出相关推荐

Kube Controller Manager 源码分析
Kube Controller Manager 源码分析 Controller Manager 在k8s 集群中扮演着中心管理的角色,它负责Deployment, StatefulSet, Repli ...
Nginx 403 Forbidden 排错记录汇总
记录一当你将 nginx 作为 web server 的时候,403 错误主要是下面两条原因: 一.所有者对目录没有写的权限. 此时可用 chmod 777 目录名先完全放开权限,如果问题解决,则 ...
kubernetes故障现场一之Orphaned pod
系列目录问题描述:周五写字楼整体停电,周一再来的时候发现很多pod的状态都是Terminating,经排查是因为测试环境kubernetes集群中的有些节点是PC机,停电后需要手动开机才能起来.起来 ...
kubelet报错：Orphaned pod “$Pod“ found, but volume paths are still present on disk
1.场景: 容器一直处在 ContainerCreating状态 $ kubectl get pod -n liantiao | grep pushserver pushserver-69cc66c6 ...
k8s关于Orphaned pod ＜pod_id＞ found，volume paths are still present on disk 的解决方法
问题描述因k8s节点异常关机导致启动后业务Pod重新部署,关机之前的Pod状态已被删除,今天在查看日志时发现在异常关机之前的集群节点Pod是非正常移除的,一直刷报错信息:如下: 问题排查查看系统日 ...
Orphaned pod found - but volume paths are still present on disk的处理
1. 现象一个问题发现集群内节点非正常移除,被调度为not ready 2. 问题排查查看kubelet日志 # journal -u kubelet -f Jan 21 16:45:44 loc ...
Kubernetes源码阅读笔记——Controller Manager（之三）
前一篇文章中,我们探索了Informer工作的大致逻辑,提到了添加回调函数部分包含了三块,即:Informer的创建.函数调用的逻辑.以及回调函数本身.前两块已在前文谈到过,下面我们来看看第三块,即回 ...
Kubernetes Controller Manager 工作原理
原文连接:https://blog.ihypo.net/15763910382218.html 本文基于对 Kubernetes v1.16 的源码阅读,文章有一定的源码,但我会通过配图尽量描述清晰 ...
kubernetes 核心组件之 Controller Manager
文章目录 Controller Manager 简介 Controller Manager 架构设计 Reflactor Informer Controller Replication Control ...

rook 排错记录 + Orphaned pod found kube-controller-manager的日志输出

rook 排错记录 + Orphaned pod found kube-controller-manager的日志输出相关推荐

最新文章

热门文章