kubelet 无法启动排查

问题描述

使用ansible安装Kubernetes，最后出现如所示报错，提示kubelet启动异常

TASK [kube-node : 轮询等待kubelet启动] ******************************************************************************************************************************
fatal: [192.168.10.52]: FAILED! => {"attempts": 4, "changed": true, "cmd": "systemctl is-active kubelet.service", "delta": "0:00:00.006796", "end": "2023-02-01 22:30:10.756458", "msg": "non-zero return code", "rc": 3, "start": "2023-02-01 22:30:10.749662", "stderr": "", "stderr_lines": [], "stdout": "activating", "stdout_lines": ["activating"]}
fatal: [192.168.10.51]: FAILED! => {"attempts": 4, "changed": true, "cmd": "systemctl is-active kubelet.service", "delta": "0:00:00.010879", "end": "2023-02-01 22:30:10.859450", "msg": "non-zero return code", "rc": 3, "start": "2023-02-01 22:30:10.848571", "stderr": "", "stderr_lines": [], "stdout": "activating", "stdout_lines": ["activating"]}PLAY RECAP **********************************************************************************************************************************************************
192.168.10.51              : ok=50   changed=30   unreachable=0    failed=1    skipped=1    rescued=0    ignored=0
192.168.10.52              : ok=49   changed=30   unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

问题排查

检查kubelet状态，显示没启动成功

使用journalctl -u kubelet --no-pager 查看启动报错日志

Dec 07 23:50:21 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[24786]: E1207 23:50:21.347929   24786 remote_runtime.go:168] "Version from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
Dec 07 23:50:21 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[24786]: E1207 23:50:21.348041   24786 kuberuntime_manager.go:225] "Get runtime version failed" err="get remote runtime typed version failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
Dec 07 23:50:21 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[24786]: Error: failed to run Kubelet: failed to create kubelet: get remote runtime typed version failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService

依据报错应该是containerd的问题，确认containerd状态

应该和配置文件/etc/containerd/config.toml中的disabled_plugins = ["cri"]有关，详情参见https://github.com/containerd/containerd/issues/4581

移除/etc/containerd/config.toml配置文件

grep "disabled_plugins" /etc/containerd/config.toml
mv /etc/containerd/config.toml /tmp/

重启 kubelet 成功

问题原因

https://github.com/containerd/containerd/issues/4581、

解决办法

mv /etc/containerd/config.toml /tmp
systemctl restart containerd
systemctl restart kubelet

kubelet 无法启动排查相关推荐

解决Kubelet Pod启动CreatePodSandbox或RunPodSandbox异常方法
解决Kubelet Pod启动CreatePodSandbox或RunPodSandbox异常方法参考文章: (1)解决Kubelet Pod启动CreatePodSandbox或RunPodSan ...
linux错误代码255,kubelet 服务启动失败, 错误代码 255
kubelet 服务启动失败, 错误代码 255 用 kubeadm 的方法安装 kubelet 后, 运行 systemctl status kubelet 发现 kubelet 服务启动失败, 错 ...
状态代码255 必须重启服务器,kubelet服务启动失败，错误代码255
用kubeadm的方法安装kubelet后,运行systemctl status kubelet 发现kubelet服务启动失败,错误代码255. kubelet.service: main proc ...
《Dive into Windbg系列》Explorer无法启动排查
转载自:https://www.anquanke.com/post/id/179748 <Dive into Windbg>是一系列关于如何理解和使用Windbg的文章,主要涵盖三个方面: ...
手动安装k8s,但kubelet.service 启动失败
k8s的kubelet.service 启动失败 #systemctl status kubelet.service ● kubelet.service - Kubernetes API Server ...
ElasticSearch启动该正常无法连接或无法正常启动排查方案
ElasticSearch启动该正常无法连接或无法正常启动排查方案文章目录 ElasticSearch启动该正常无法连接或无法正常启动排查方案一.8.2版本及以上默认https链接二. JDK版 ...
IPguard服务器无法启动排查
1.服务器启动报错访问IPguard路径错误,权限不够,修改文件夹权限,everyone 完全控制 2.新安装的IPguard服务器无法启动,查看数据库,打开数据库,安全性-登录名-NT AUTHOR ...
kubelet启动失败_《蹲坑学kubernetes》之10-1：kubelet原理详解
在kubernetes集群中,每个Node节点上都运行一个Kubelet服务进程,默认监听10250端口,接收并执行Master发来的指令,管理Pod及Pod中的容器.每个Kubelet进程会在API ...
kubelet启动失败_kubelet 架构浅析
一.概要 kubelet 是运行在每个节点上的主要的"节点代理",每个节点都会启动 kubelet进程,用来处理 Master 节点下发到本节点的任务,按照 PodSpec 描述来 ...