ingress-nginx部署状态为CrashLoopBackOff 问题排查

ingress-nginx部署状态为CrashLoopBackOff 问题排查，说起来这个问题挺坑的，kubernetes集群部署的时候，没有开启ipvs转发，所以会有这个问题

系统版本：CentOs8.1
k8s版本：v1.21.3
containerd版本：ctr containerd.io 1.4.3

ingress部署后状态为CrashLoopBackOff ,不停的重启

[root@ck8s1 ingress]# kubectl get pod -n ingress-nginx
NAME                                        READY   STATUS             RESTARTS   AGE
default-http-backend-7b6d9847f6-crgs8       1/1     Running            0          46m
nginx-ingress-controller-7bbb744996-rd9d6   0/1     CrashLoopBackOff   19         46m

describe查看信息如下：

[root@ck8s1 ingress]# kubectl describe pod nginx-ingress-controller-7bbb744996-rd9d6 -n ingress-nginx
Name:         nginx-ingress-controller-7bbb744996-rd9d6
Namespace:    ingress-nginx
Priority:     0
Node:         ck8s2/192.168.43.152
Start Time:   Sun, 15 Aug 2021 12:06:08 +0800
Labels:       app.kubernetes.io/name=ingress-nginxapp.kubernetes.io/part-of=ingress-nginxpod-template-hash=7bbb744996
Annotations:  cni.projectcalico.org/containerID: f59aa02e0654127673fc968533873d70d3b172c6f9cea0f7cab027efc662ee46cni.projectcalico.org/podIP: 192.168.135.151/32cni.projectcalico.org/podIPs: 192.168.135.151/32prometheus.io/port: 10254prometheus.io/scrape: true
Status:       Running
IP:           192.168.135.151
IPs:IP:           192.168.135.151
Controlled By:  ReplicaSet/nginx-ingress-controller-7bbb744996
Containers:nginx-ingress-controller:Container ID:  containerd://f0f3c6cb2aafb5bc7474fc44c7b680aede4a481ef0b0c8826a71af88ffa0e663Image:         registry.aliyuncs.com/kubernetes/nginx-ingress-controller:0.20.0Image ID:      registry.aliyuncs.com/kubernetes/nginx-ingress-controller@sha256:3f06079f7727b2fb7ad5c97d8152eb622ae504674395dfa71fda7ce315aaaf30Ports:         80/TCP, 443/TCPHost Ports:    0/TCP, 0/TCPArgs:/nginx-ingress-controller--default-backend-service=$(POD_NAMESPACE)/default-http-backend--configmap=$(POD_NAMESPACE)/nginx-configuration--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services--udp-services-configmap=$(POD_NAMESPACE)/udp-services--publish-service=$(POD_NAMESPACE)/ingress-nginx--annotations-prefix=nginx.ingress.kubernetes.ioState:          WaitingReason:       CrashLoopBackOffLast State:     TerminatedReason:       ErrorExit Code:    143Started:      Sun, 15 Aug 2021 12:44:59 +0800Finished:     Sun, 15 Aug 2021 12:45:38 +0800Ready:          FalseRestart Count:  17Liveness:       http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3Readiness:      http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3Environment:POD_NAME:       nginx-ingress-controller-7bbb744996-rd9d6 (v1:metadata.name)POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)Mounts:/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mj82d (ro)
Conditions:Type              StatusInitialized       True Ready             False ContainersReady   False PodScheduled      True
Volumes:kube-api-access-mj82d:Type:                    Projected (a volume that contains injected data from multiple sources)TokenExpirationSeconds:  3607ConfigMapName:           kube-root-ca.crtConfigMapOptional:       <nil>DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300snode.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:Type     Reason     Age                    From               Message----     ------     ----                   ----               -------Normal   Scheduled  43m                    default-scheduler  Successfully assigned ingress-nginx/nginx-ingress-controller-7bbb744996-rd9d6 to ck8s2Normal   Created    42m (x2 over 43m)      kubelet            Created container nginx-ingress-controllerNormal   Started    42m (x2 over 43m)      kubelet            Started container nginx-ingress-controllerNormal   Pulled     41m (x3 over 43m)      kubelet            Container image "registry.aliyuncs.com/kubernetes/nginx-ingress-controller:0.20.0" already present on machineWarning  Unhealthy  41m (x10 over 43m)     kubelet            Readiness probe failed: Get "http://192.168.135.151:10254/healthz": dial tcp 192.168.135.151:10254: connect: connection refusedNormal   Killing    41m (x2 over 42m)      kubelet            Container nginx-ingress-controller failed liveness probe, will be restartedWarning  Unhealthy  22m (x34 over 42m)     kubelet            Liveness probe failed: Get "http://192.168.135.151:10254/healthz": dial tcp 192.168.135.151:10254: connect: connection refusedWarning  BackOff    2m56s (x147 over 40m)  kubelet            Back-off restarting failed container
[root@ck8s1 ingress]#

查看pod日志

[root@ck8s1 ingress]# kubectl logs nginx-ingress-controller-7bbb744996-rd9d6 -n ingress-nginx
-------------------------------------------------------------------------------
NGINX Ingress controllerRelease:    0.20.0Build:      git-e8d8103Repository: https://github.com/kubernetes/ingress-nginx.git
-------------------------------------------------------------------------------nginx version: nginx/1.15.5
W0815 04:44:59.166129       8 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0815 04:44:59.166490       8 main.go:196] Creating API client for https://10.10.0.1:443

发现是443端口的问题，测试端口不通

[root@ck8s1 ingress]# telnet 10.0.0.1 443
Trying 10.0.0.1...

1，解决方法：
修改kube-proxy配置把流量转发该成ipvs模式

2，编辑配置：
kubectl edit configmap kube-proxy -n kube-system
将 mode: “” 为空改成 mode: “ipvs”
将 masqueradeAll: null 改成 masqueradeAll: yes

3，删除所有kube-proxy的pod

 kubectl get pod -n kube-system | grep kube-proxy | awk '{system(" kubectl delete pod "$1" -n kube-system")}'

4，校验
kubectl logs kube-proxy-xxx -n kube-system 检查是否日志出现Using ipvs Proxier

5，重新部署ingress-nginx

[root@ck8s1 ingress]# kubectl get pod -n ingress-nginx
NAME                                        READY   STATUS    RESTARTS   AGE
default-http-backend-7b6d9847f6-5jt5b       1/1     Running   0          13m
nginx-ingress-controller-7bbb744996-2c7tp   1/1     Running   0          13m

启动正常，没有报错

ingress-nginx部署状态为CrashLoopBackOff 问题排查相关推荐

Kubernetes K8S 1.20部署Ingress nginx 0.30
主机配置规划服务器名称(hostname) 系统版本配置内网IP 外网IP(模拟) k8s-master CentOS7.7 2C/4G/20G 172.16.1.110 10.0.0.110 ...
nginx 400状态码排查
最近,发现主站nginx的log中有很多400的错误,每天有几千万条,故决定对产生400错误的原因进行排查.分析nginx log,发现这种无效的400请求,总是在一个正常访问之后产生,一般出现一个或 ...
Nginx Ingress Controller 部署
概述本次实践的主要目的就是将入口统一,不再通过 LoadBalancer 等方式将端口暴露出来,而是使用 Ingress 提供的反向代理负载均衡功能作为我们的唯一入口.通过以下步骤操作仔细体会. 注 ...
网关流控利器：结合 AHAS 实现 Ingress/Nginx 流量控制
作者:涂鸦微服务的稳定性一直是开发者非常关注的话题.随着业务从单体架构向分布式架构演进以及部署方式的变化,服务之间的依赖关系变得越来越复杂,业务系统也面临着巨大的高可用挑战.应用高可用服务 AHAS ...
深入浅析nginx部署及简单优化
Nginx是lgor Sysoev为俄罗斯访问量第二的rambler.ru站点设计开发的.本文重点给大家介绍nginx部署及简单优化方案,感兴趣的朋友一起看看吧 Nginx是lgor Sysoev为俄 ...
Nginx部署及综合应用
介绍 Nginx是一款轻量级的HTTP服务器软件,由俄罗斯的Igor Sysoev开发.它能够支持高达50000个并发连接数的响应,拥有强大的静态资源处理能力,运行稳定,并且系统资源消耗非常低,现已逐 ...
django+uwsgi+nginx部署
最近线上部署运维工单系统,使用django+uwsgi+nginx部署,发现这篇文章,很详细,觉得不错,转载一下: 写了那么长时间的django,一直没跟nginx结合一下,访问的时候老是得加端口号, ...
vps实现私人代码托管并用nginx部署hexo
个人博客原本我的博客是通过github pages搭建的,但由于一些众所周知的原因,即使套上了CDN依旧是访问状态令人堪忧,经常会造成各种各样的问题,并且由于不存在服务器也不好进行进一步的管理,更不 ...
Ubuntu配置Nginx部署Vue SPA项目
Ubuntu配置Nginx部署Vue SPA项目文章记录了部署Vue SPA项目后可从ip访问的过程,因域名访问需要备案暂未尝试,后续若实行亦会更新在此处. 系统为Ubuntu18.04. Ngin ...

ingress-nginx部署状态为CrashLoopBackOff 问题排查

ingress-nginx部署状态为CrashLoopBackOff 问题排查相关推荐

最新文章

热门文章