CNI failed to retrieve network namespace path

CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container

我解决这个问题进行了以下操作：

重启Calico（没有变化）
重启Docker和Kubelet（出现了新的OOM的提示）
更改Pod资源配额，问题解决
再次调小资源配额，问题复现

因此标题所述问题跟资源配额太小有关。但是Pod的日志，Calico等网络插件的信息均没有直接指出问题所在。

Pod定义：

apiVersion: v1
kind: Pod
metadata:name: downward
spec:containers:- name: mainimage: busybox:1.28.4imagePullPolicy: IfNotPresentcommand: ["sleep", "9999999"]resources:requests:cpu: 15mmemory: 100Kilimits:cpu: 100mmemory: 4Mienv:- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: POD_IPvalueFrom:fieldRef:fieldPath: status.podIP- name: NODE_NAMEvalueFrom:fieldRef:fieldPath: spec.nodeName- name: SERVICE_ACCOUNTvalueFrom:fieldRef:fieldPath: spec.serviceAccountName- name: CONTAINER_CPU_REQUEST_MILLICORESvalueFrom:resourceFieldRef:resource: requests.cpudivisor: 1m- name: CONTAINER_MEMORY_LIMIT_KIBIBYTESvalueFrom:resourceFieldRef:resource: limits.memorydivisor: 1Ki

Pod日志：

  Warning  FailedCreatePodSandBox  0s                 kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ba50e2ad6faf226a6680e2ba61179c626825141034b63611265463d71de57ae1" network for pod "downward": networkPlugin cni failed to set up pod "downward_kubernetes-dashboard" network: CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "ba50e2ad6faf226a6680e2ba61179c626825141034b63611265463d71de57ae1"

排查过程：
在网上查找了很多资料都不得其解，当然本人对K8S和Calico也不熟悉。
在stackoverflow上查到类似问题，但是解答得不清楚，为：

I solve this problem. The cause was argo Cd prune setting

在Redhat bugzilla中，没看清，说不是一个问题？
在stackoverflow中还看到一个帖子，这么说的：

Ensure that /etc/cni/net.d and its /opt/cni/bin friend both exist and are correctly populated with the CNI configuration files and binaries on all Nodes. For flannel specifically, one might make use of the flannel cni repo
Don’t overlook the are correctly populated part. For that /var/lib/calico/nodename error specifically, be careful if you are using a DaemonSet to configure calico: k8s may try to schedule that frontend Pod on the Node before the DaemonSet has finished configuring it. In that specific circumstance, just deleting all the Pods from the Node after calico is successfully configured will cure that sandbox problem

找不到原因，重启kubelet和docker后，发现新增了如下错误：

  Warning  FailedMount             77s (x5 over 88s)      kubelet            MountVolume.SetUp failed for volume "kube-api-access-rj9w6" : failed to fetch token: Post "https://10.0.12.10:6443/api/v1/namespaces/kubernetes-dashboard/serviceaccounts/default/token": dial tcp 10.0.12.10:6443: connect: connection refusedNormal   SandboxChanged          52s (x10 over 68s)     kubelet            Pod sandbox changed, it will be killed and re-created.Warning  FailedCreatePodSandBox  52s (x9 over 67s)      kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "downward": Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: container init was OOM-killed (memory limit too low?): unknown

难道是因为资源限制太小的缘故？尝试将资源限制调整为：

    resources:requests:cpu: 15mmemory: 100Milimits:cpu: 100mmemory: 400Mi

Pod正常启动了！

CNI failed to retrieve network namespace path相关推荐

seting the network namespace failed: Invalid argument
使用Quantum的时候,会创建ip netns,当删除虚机或者删除子网的时候一般在l3的日志里可以看到这个错误,这个应该是个bug 原因是因为在删除之前,就把虚拟网络的namespace读权限取消 ...
Linux namespace之：network namespace
理解network namespace network namespace用来隔离网络环境,「在network namespace中,网络设备.端口.套接字.网络协议栈.路由表.防火墙规则等都是独立的 ...
Kubernetes network namespace位置在哪里？
linux network namespace 通常情况下/var/run/netns存放network namespace,但是docker没有将数据同步到linux runtime. 因此ip n ...
如何解决hadoop文件无法浏览问题“Failed to retrieve data from /webhdfs/v1/?op=LISTSTATUS: Server Error“
搭建好了hadoop环境后,namenode.datanode看着都是正常的.但是当点到browser the file system时,结果报错了,报错的内容是"Failed to ret ...
linux网络命名空间详解,Linux Network Namespace (netns) 详解
Linux Network Namespace (netns) 详解 Network Namespace (以下简称netns)是Linux内核提供的一项实现网络隔离的功能,它能隔离多个不同的网络空间 ...
Cygwin运行nutch报错：Failed to set permissions of path
2019独角兽企业重金招聘Python工程师标准>>> 错误信息: Exception in thread "main" java.io.IOException: ...
linux 的ip 设置lo_linux网络虚拟化： network namespace 简介及实验
namespace(命名空间)和cgroup是软件容器化(想想Docker)趋势中的两个主要内核技术.简单来说,cgroup是一种对进程进行统一的资源监控和限制,它控制着你可以使用多少系统资源(CPU ...
Flutter web问题：Failed to load network image
Flutter web问题:Failed to load network image 我的解决办法: flutter build web --release --web-renderer html f ...
如何定位“Operating system error 32(failed to retrieve text for this error. Reason: 15105)”错误中被占用的文件...
之前在这篇"Operating system error 32(failed to retrieve text for this error. Reason: 15105)"博 ...

CNI failed to retrieve network namespace path

CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container

CNI failed to retrieve network namespace path相关推荐

最新文章

热门文章