k8s中pod sandbox创建失败failed to start sandbox container
2024-05-13 12:43:05
背景
今天在k8s更新服务时,发现pod启动失败,报错failed to start sandbox container,如下所示:
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal Scheduled 28m default-scheduler Successfully assigned kube-system/k8s-proxy-7wkt4 to tj1-staging-com-ocean007-201812.kscnWarning FailedCreatePodSandBox 28m (x13 over 28m) kubelet, tj1-staging-com-ocean007-201812.kscn Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "k8s-proxy-7wkt4": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:297: getting the final child's pid from pipe caused \"EOF\"": unknownNormal SandboxChanged 3m19s (x1364 over 28m) kubelet, tj1-staging-com-ocean007-201812.kscn Pod sandbox changed, it will be killed and re-created.
分析
sandbox 创建失败只是表象,是宿主机其他异常导致的,一般是(cpu,diskio,mem)导致的.
首先,上节点看kubelet,docker有无异常,日志没有明显错误,通过top看到docker cpu占用非常高
[root@tj1-staging-com-ocean007-201812 ~]# toptop - 17:55:00 up 265 days, 3:41, 1 user, load average: 10.71, 11.34, 10.76
Tasks: 816 total, 5 running, 811 sleeping, 0 stopped, 0 zombie
%Cpu(s): 24.0 us, 34.5 sy, 0.0 ni, 41.4 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 65746380 total, 20407940 free, 11007040 used, 34331400 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 49134416 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 115483 root 20 0 3965212 273188 34564 S 489.7 0.4 382260:40 dockerd
1367523 root 20 0 18376 2972 2716 R 66.9 0.0 20163:45 bash
1367487 root 20 0 11856 5616 4512 S 54.0 0.0 16748:26 containerd-shim
3200169 root 20 0 1300 4 0 R 53.3 0.0 14913:49 sh
2429952 root 20 0 1300 4 0 S 49.3 0.0 9620:56 sh
3200130 root 20 0 9392 4756 3884 S 47.7 0.0 13417:30 containerd-shim
3718475 root 20 0 1300 4 0 R 47.4 0.0 8600:20 sh
3718440 root 20 0 10736 5516 4512 S 42.1 0.0 7575:31 containerd-shim
2429917 root 20 0 11856 5556 4512 S 40.1 0.0 8313:22 containerd-shim
3205493 root 20 0 3775924 230996 66704 S 18.9 0.4 2559:07 kubelet 1 root 20 0 195240 157000 3932 S 7.9 0.2 1417:46 systemd 804 dbus 20 0 30308 6460 2464 S 1.7 0.0 462:18.84 dbus-daemon
1011737 root 20 0 277656 122788 18428 S 1.3 0.2 768:03.00 cadvisor 115508 root 20 0 7139200 32896 24288 S 1.0 0.1 662:25.27 containerd 806 root 20 0 24572 3060 2480 S 0.7 0.0 171:22.52 systemd-logind 511080 root 0 -20 2751348 52552 15744 S 0.7 0.1 178:27.51 sagent
1102507 root 20 0 11792 7292 4512 S 0.7 0.0 23:36.37 containerd-shim
1272223 root 20 0 164800 5296 3824 R 0.7 0.0 0:00.38 top
2866292 root 20 0 5045000 1.983g 3080 S 0.7 3.2 230:09.47 redis
同时, cpu system异常高.
%Cpu(s): 24.0 us, 34.5 sy, 0.0 ni, 41.4 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
按照以前的经验,一般是由某些容器引起的,通过top看到个别sh进程占用cpu较高.
通过ps看到进程居然是个死循环
[root@tj1-staging-com-ocean007-201812 ~]# ps -ef |grep 1367523
root 1287628 1247781 0 17:55 pts/1 00:00:00 grep --color=auto 1367523
root 1367523 1367504 72 Feb28 ? 14-00:04:17 /bin/bash -c while true; do echo hello; done
通过/proc/pid/cgroup找到对应容器
# cat /proc/1367523/cgroup
11:freezer:/kubepods/besteffort/pod55d3adf2-67f7-11ea-93f2-246e968203b8/29842d5544b701dbb5ff647dba19bb4ebec821edc6ee1ffbd7aeee58fa5038fd
10:devices:/kubepods/besteffort/pod55d3adf2-67f7-11ea-93f2-246e968203b8/29842d5544b701dbb5ff647dba19bb4ebec821edc6ee1ffbd7aeee58fa5038fd
9:hugetlb:/kubepods/besteffort/pod55d3adf2-67f7-11ea-93f2-246e968203b8/29842d5544b701dbb5ff647dba19bb4ebec821edc6ee1ffbd7aeee58fa5038fd
8:blkio:/kubepods/besteffort/pod55d3adf2-67f7-11ea-93f2-246e968203b8/29842d5544b701dbb5ff647dba19bb4ebec821edc6ee1ffbd7aeee58fa5038fd
7:memory:/kubepods/besteffort/pod55d3adf2-67f7-11ea-93f2-246e968203b8/29842d5544b701dbb5ff647dba19bb4ebec821edc6ee1ffbd7aeee58fa5038fd
6:perf_event:/kubepods/besteffort/pod55d3adf2-67f7-11ea-93f2-246e968203b8/29842d5544b701dbb5ff647dba19bb4ebec821edc6ee1ffbd7aeee58fa5038fd
5:cpuset:/kubepods/besteffort/pod55d3adf2-67f7-11ea-93f2-246e968203b8/29842d5544b701dbb5ff647dba19bb4ebec821edc6ee1ffbd7aeee58fa5038fd
4:pids:/kubepods/besteffort/pod55d3adf2-67f7-11ea-93f2-246e968203b8/29842d5544b701dbb5ff647dba19bb4ebec821edc6ee1ffbd7aeee58fa5038fd
3:net_cls,net_prio:/kubepods/besteffort/pod55d3adf2-67f7-11ea-93f2-246e968203b8/29842d5544b701dbb5ff647dba19bb4ebec821edc6ee1ffbd7aeee58fa5038fd
2:cpu,cpuacct:/kubepods/besteffort/pod55d3adf2-67f7-11ea-93f2-246e968203b8/29842d5544b701dbb5ff647dba19bb4ebec821edc6ee1ffbd7aeee58fa5038fd
1:name=systemd:/kubepods/besteffort/pod55d3adf2-67f7-11ea-93f2-246e968203b8/29842d5544b701dbb5ff647dba19bb4ebec821edc6ee1ffbd7aeee58fa5038fd
找到对应容器
docker ps | grep 29842d554
清理完相关pod后,系统恢复正常
top - 18:25:57 up 265 days, 4:12, 1 user, load average: 1.05, 1.24, 4.02
Tasks: 769 total, 1 running, 768 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.7 us, 0.9 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 65746380 total, 22106960 free, 10759860 used, 32879560 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 49401576 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3205493 root 20 0 3775924 229844 66704 S 9.9 0.3 2563:18 kubelet 115483 root 20 0 3965468 249124 34564 S 7.9 0.4 382323:36 dockerd 1 root 20 0 195240 157000 3932 S 6.3 0.2 1419:48 systemd 804 dbus 20 0 30308 6460 2464 S 2.0 0.0 462:51.51 dbus-daemon
3085322 root 20 0 12.045g 1.578g 19028 S 1.3 2.5 767:51.19 java 115508 root 20 0 7139200 32264 24288 S 1.0 0.0 662:42.18 containerd 511080 root 0 -20 2751348 42116 15744 S 1.0 0.1 178:44.79 sagent
1011737 root 20 0 277656 111836 18428 S 1.0 0.2 768:49.01 cadvisor
1523167 root 20 0 164800 5436 4012 R 0.7 0.0 0:00.04 top
3199459 root 20 0 1554708 43668 9496 S 0.7 0.1 28:50.60 falcon-agent 7 root 20 0 0 0 0 S 0.3 0.0 619:07.64 rcu_sched 806 root 20 0 24572 3060 2480 S 0.3 0.0 171:33.69 systemd-logind 11921 root 20 0 94820 20480 5840 S 0.3 0.0 1402:42 consul 575838 root 20 0 411464 17092 7364 S 0.3 0.0 15:16.25 python 856593 root 20 0 1562392 37912 9612 S 0.3 0.1 21:34.23 falcon-agent 931957 33 20 0 90728 3392 1976 S 0.3 0.0 0:51.23 nginx
1212186 root 20 0 0 0 0 S 0.3 0.0 0:01.12 kworker/14:1
1726228 root 20 0 9392 4496 3808 S 0.3 0.0 0:00.67 containerd-shim
1887128 root 20 0 273160 7932 3128 S 0.3 0.0 46:05.23 redis-server
2788111 root 20 0 273160 6300 3080 S 0.3 0.0 25:18.55 redis-server
3199297 root 20 0 1563160 44812 9624 S 0.3 0.1 31:13.73 falcon-agent
总结
sandox创建失败的原因是各种各样的, 如[memory设置错误触发的异常][1],[dockerd异常][2].
针对此处问题是由于某些测试pod通过while true; do echo hello; done启动,死循环一直echo hello产生大量read()系统调用,所在cpu飙升.多个类似pod导致系统非常繁忙,无法正常处理其他请求.
此类问题不容易在pod创建时直接检测到,只能通过添加物理节点相关报警(dockerd cpu使用率, node cpu.sys使用率)及时发现问题.
引用
[1] https://github.com/kubernetes/kubernetes/issues/56996
[2] https://plugaru.org/2018/05/21/pod-sandbox-changed-it-will-be-killed-and-re-created/
k8s中pod sandbox创建失败failed to start sandbox container相关推荐
- 图解k8s中pod的创建流程
什么是pod Pod 是 k8s 系统中可以创建和管理的最小单元,是资源对象模型中由用户创建或部署的最小资源对象模型,也是在 k8s 上运行容器化应用的资源对象,其他的资源对象都是用来支撑或者扩展 P ...
- K8S中pod健康状态的检查
对于Pod的健康状态检测,kubernetes提供了两类探针(Probe)来实现对k8s中Pod的健康状态进行检测 什么是 Container Probes 通过k8s的架构图,我们可以发现,每个No ...
- 【好文收藏】k8s中Pod 无法正常解析域名：部署 DNS 调试工具排查
k8s 中 Pod 无法正常解析域名:部署 DNS 调试工具排查 问题描述 最近将 Kubernetes 升级到 1.18.1 版本,不过升级完后,查看工作节点的部分 Pod 无法启动,查看消息全是 ...
- 关于K8s中Pod调度[选择器,指定节点,主机亲和性]方式和节点[coedon,drain,taint]标记的Demo
写在前面 嗯,整理K8s中pod调度相关笔记,这里分享给小伙伴 博文内容涉及: kube-scheduler组件的简述 Pod的调度(选择器.指定节点.主机亲和性)方式 节点的coedon与drain ...
- k8s中pod的重启策略和健康检查
目录 k8s中pod的重启策略 pod中一共有以下三个重启策略(restartPolicy) 健康检查: 健康检查类型 支持的检查方法: 检查示例 其他检查方式示例 k8s中pod的重启策略 pod中 ...
- K8s中Pod健康检查源代码分析
了解k8s中的Liveness和Readiness Liveness: 表明是否容器正在运行.如果liveness探测为fail,则kubelet会kill掉容器,并且会触发restart设置的策略 ...
- 查看k8s中Pod里容器的数量和名称
查看Pod里容器的名称 初始化一个包含两个容器的Pod(tomcat和nginx),其中文件名为ini-pod.yaml apiVersion: v1 kind: Pod metadata:name: ...
- k8s中Pod域名解析失败
原文 系统环境 CoreDNS 版本:1.6.7 Docker 版本:19.03.8 Kubernetes 版本:1.18.4 操作系统版本:CentOS 7.4 CentOS 内核版本:3.10.0 ...
- K8s中Pod生命周期和重启策略
这周项目组提到了POD 的健康检查,自己在春节之中刚好度过,现在认真复习一下 复习以下<K8s权威指南>中第三章节深入理解pod中的关于POD 声明周期和重启策略的内容 一.POD状态: ...
最新文章
- Linux Kernel TCP/IP Stack — L3 Layer — netfilter 框架
- Redis Python 客户端
- 修改系统UIAlertAction的按钮颜色
- BIETLOLTP之概念熟悉
- 轻松几步搞定SSH连接Git配置
- 最强阿里面试126题：数据结构+并发编程+Redis+设计模式+微服务
- java 课后习题 Vector类的 初使用
- QT之 Hello World
- 在PyQt中构建 Python 菜单栏、菜单和工具栏
- 简单nginx+tomca负载均衡
- 安装部署shipyard
- iperf3 网络探测详解（android、iOS、windows）
- Hibernate（2）
- 非合作博弈篇——纳什均衡(Nash Equilibrium)
- ICMAX解析运行内存发展新趋势 LPDDR4X将会给手机带来哪些改变？
- 随机过程 Class 3 条件期望
- C语言之图书管理系统（功能齐全！！！）
- 【硬核技术文】研发绩效，AI算法的完美舞台
- 第二十三讲 常用技术标准【2021年软考-高级信息系统项目管理师】
- HEG运行错误：GetBeginAndEndLocs