1. 查看系统Event事件

kubectl describe pod <PodName> --namespace=<NAMESPACE>

该命令可以显示Pod创建时的配置定义、状态等信息和最近的Event事件，事件信息可用于排错。例如当Pod状态为Pending，可通过查看Event事件确认原因，一般原因有几种：

没有可用的Node可调度
开启了资源配额管理并且当前Pod的目标节点上恰好没有可用的资源
正在下载镜像（镜像拉取耗时太久）或镜像下载失败。

kubectl describe还可以查看其它k8s对象：NODE,RC,Service,Namespace,Secrets。

1.1. Pod

kubectl describe pod <PodName> --namespace=<NAMESPACE>

以下是容器的启动命令非阻塞式导致容器挂掉，被k8s频繁重启所产生的事件。

kubectl describe pod <PodName> --namespace=<NAMESPACE>  Events:FirstSeen LastSeen    Count   From            SubobjectPath       Reason      Message───────── ────────    ─────   ────            ─────────────       ──────      ───────7m        7m      1   {scheduler }                    Scheduled   Successfully assigned yangsc-1-0-0-index0 to 10.8.216.197m        7m      1   {kubelet 10.8.216.19}   containers{infra}   Pulled      Container image "gcr.io/kube-system/pause:0.8.0" already present on machine7m        7m      1   {kubelet 10.8.216.19}   containers{infra}   Created     Created with docker id 84f133c324d07m        7m      1   {kubelet 10.8.216.19}   containers{infra}   Started     Started with docker id 84f133c324d07m        7m      1   {kubelet 10.8.216.19}   containers{yangsc0} Started     Started with docker id 3f9f82abb1457m        7m      1   {kubelet 10.8.216.19}   containers{yangsc0} Created     Created with docker id 3f9f82abb1457m        7m      1   {kubelet 10.8.216.19}   containers{yangsc0} Created     Created with docker id fb112e4002f47m        7m      1   {kubelet 10.8.216.19}   containers{yangsc0} Started     Started with docker id fb112e4002f46m        6m      1   {kubelet 10.8.216.19}   containers{yangsc0} Created     Created with docker id 613b119d44746m        6m      1   {kubelet 10.8.216.19}   containers{yangsc0} Started     Started with docker id 613b119d44746m        6m      1   {kubelet 10.8.216.19}   containers{yangsc0} Created     Created with docker id 25cb68d1fd3d6m        6m      1   {kubelet 10.8.216.19}   containers{yangsc0} Started     Started with docker id 25cb68d1fd3d5m        5m      1   {kubelet 10.8.216.19}   containers{yangsc0} Started     Started with docker id 7d9ee8610b285m        5m      1   {kubelet 10.8.216.19}   containers{yangsc0} Created     Created with docker id 7d9ee8610b283m        3m      1   {kubelet 10.8.216.19}   containers{yangsc0} Started     Started with docker id 88b9e8d582dd3m        3m      1   {kubelet 10.8.216.19}   containers{yangsc0} Created     Created with docker id 88b9e8d582dd7m        1m      7   {kubelet 10.8.216.19}   containers{yangsc0} Pulling     Pulling image "gcr.io/test/tcp-hello:1.0.0"1m        1m      1   {kubelet 10.8.216.19}   containers{yangsc0} Started     Started with docker id 089abff050e71m        1m      1   {kubelet 10.8.216.19}   containers{yangsc0} Created     Created with docker id 089abff050e77m        1m      7   {kubelet 10.8.216.19}   containers{yangsc0} Pulled      Successfully pulled image "gcr.io/test/tcp-hello:1.0.0"6m        7s      34  {kubelet 10.8.216.19}   containers{yangsc0} Backoff     Back-off restarting failed docker container

1.2. NODE

kubectl describe node 10.8.216.20

[root@FC-43745A-10 ~]# kubectl describe node 10.8.216.20
Name:           10.8.216.20
Labels:         kubernetes.io/hostname=10.8.216.20,namespace/bcs-cc=true,namespace/myview=true
CreationTimestamp:  Mon, 17 Apr 2017 11:32:52 +0800
Phase:
Conditions:  Type      Status  LastHeartbeatTime           LastTransitionTime          Reason              Message  ────      ──────  ─────────────────           ──────────────────          ──────              ───────  Ready     True    Fri, 18 Aug 2017 09:38:33 +0800     Tue, 02 May 2017 17:40:58 +0800     KubeletReady            kubelet is posting ready status  OutOfDisk     False   Fri, 18 Aug 2017 09:38:33 +0800     Mon, 17 Apr 2017 11:31:27 +0800     KubeletHasSufficientDisk    kubelet has sufficient disk space available
Addresses:  10.8.216.20,10.8.216.20
Capacity:  cpu:       32  memory:    67323039744  pods:      40
System Info:  Machine ID:            723bafc7f6764022972b3eae1ce6b198  System UUID:           4C4C4544-0042-4210-8044-C3C04F595631  Boot ID:           da01f2e3-987a-425a-9ca7-1caaec35d1e5  Kernel Version:        3.10.0-327.28.3.el7.x86_64  OS Image:          CentOS Linux 7 (Core)  Container Runtime Version: docker://1.13.1  Kubelet Version:       v1.1.1-xxx2-13.1+79c90c68bfb72f-dirty  Kube-Proxy Version:        v1.1.1-xxx2-13.1+79c90c68bfb72f-dirty
ExternalID:         10.8.216.20
Non-terminated Pods:        (6 in total)  Namespace         Name                    CPU Requests    CPU Limits  Memory Requests Memory Limits  ─────────         ────                    ────────────    ──────────  ─────────────── ─────────────  bcs-cc            bcs-cc-api-0-0-1364-index0      1 (3%)      1 (3%)      4294967296 (6%) 4294967296 (6%)  bcs-cc            bcs-cc-api-0-0-1444-index0      1 (3%)      1 (3%)      4294967296 (6%) 4294967296 (6%)  fw                fw-demo2-0-0-1519-index0        1 (3%)      1 (3%)      4294967296 (6%) 4294967296 (6%)  myview            myview-api-0-0-1362-index0      1 (3%)      1 (3%)      4294967296 (6%) 4294967296 (6%)  myview            myview-api-0-0-1442-index0      1 (3%)      1 (3%)      4294967296 (6%) 4294967296 (6%)  qa-ts-dna         ts-dna-console3-0-0-1434-index0     1 (3%)      1 (3%)      4294967296 (6%) 4294967296 (6%)
Allocated resources:  (Total limits may be over 100%, i.e., overcommitted. More info: http://releases.k8s.io/HEAD/docs/user-guide/compute-resources.md)  CPU Requests  CPU Limits  Memory Requests     Memory Limits  ────────────  ──────────  ───────────────     ─────────────  6 (18%)   6 (18%)     25769803776 (38%)   25769803776 (38%)
No events.

1.3. RC

kubectl describe rc mytest-1-0-0 --namespace=test

[root@FC-43745A-10 ~]# kubectl describe rc mytest-1-0-0 --namespace=test
Name:       mytest-1-0-0
Namespace:  test
Image(s):   gcr.io/test/mywebcalculator:1.0.1
Selector:   app=mytest,appVersion=1.0.0
Labels:     app=mytest,appVersion=1.0.0,env=ts,zone=inner
Replicas:   1 current / 1 desired
Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
No volumes.
Events:  FirstSeen LastSeen    Count   From                SubobjectPath   Reason          Message  ───────── ────────    ─────   ────                ─────────────   ──────          ───────  20h       19h     9   {replication-controller }           FailedCreate        Error creating: Pod "mytest-1-0-0-index0" is forbidden: limited to 10 pods  20h       17h     7   {replication-controller }           FailedCreate        Error creating: pods "mytest-1-0-0-index0" already exists  20h       17h     4   {replication-controller }           SuccessfulCreate    Created pod: mytest-1-0-0-index0

1.4. NAMESPACE

kubectl describe namespace test

[root@FC-43745A-10 ~]# kubectl describe namespace test
Name:   test
Labels: <none>
Status: Active  Resource Quotas  Resource       Used        Hard  ---            ---     ---  cpu            5       20  memory         1342177280  53687091200  persistentvolumeclaims 0       10  pods           4       10  replicationcontrollers 8       20  resourcequotas     1       1  secrets        3       10  services       8       20  No resource limits.

1.5. Service

kubectl describe service xxx-containers-1-1-0 --namespace=test

[root@FC-43745A-10 ~]# kubectl describe service xxx-containers-1-1-0 --namespace=test
Name:           xxx-containers-1-1-0
Namespace:      test
Labels:         app=xxx-containers,appVersion=1.1.0,env=ts,zone=inner
Selector:       app=xxx-containers,appVersion=1.1.0
Type:           ClusterIP
IP:         10.254.46.42
Port:           port-dna-tcp-35913  35913/TCP
Endpoints:      10.0.92.17:35913
Port:           port-l7-tcp-8080    8080/TCP
Endpoints:      10.0.92.17:8080
Session Affinity:   None
No events.

2. 查看容器日志

1、查看指定pod的日志

kubectl logs <pod_name>kubectl logs -f <pod_name> #类似tail -f的方式查看

2、查看上一个pod的日志

kubectl logs -p <pod_name>

3、查看指定pod中指定容器的日志

kubectl logs <pod_name> -c <container_name>

4、kubectl logs --help

[root@node5 ~]# kubectl logs --help
Print the logs for a container in a pod. If the pod has only one container, the container name is optional.
Usage:  kubectl logs [-f] [-p] POD [-c CONTAINER] [flags]
Aliases:  logs, log  Examples:
# Return snapshot logs from pod nginx with only one container
$ kubectl logs nginx
# Return snapshot of previous terminated ruby container logs from pod web-1
$ kubectl logs -p -c ruby web-1
# Begin streaming the logs of the ruby container in pod web-1
$ kubectl logs -f -c ruby web-1
# Display only the most recent 20 lines of output in pod nginx
$ kubectl logs --tail=20 nginx
# Show all logs from pod nginx written in the last hour
$ kubectl logs --since=1h nginx

3. 查看k8s服务日志

3.1. journalctl

在Linux系统上systemd系统来管理kubernetes服务，并且journal系统会接管服务程序的输出日志，可以通过systemctl status 或journalctl -u -f来查看kubernetes服务的日志。

其中kubernetes组件包括：

k8s组件	涉及日志内容	备注
kube-apiserver
kube-controller-manager	Pod扩容相关或RC相关
kube-scheduler	Pod扩容相关或RC相关
kubelet	Pod生命周期相关：创建、停止等
etcd

3.2. 日志文件

也可以通过指定日志存放目录来保存和查看日志

--logtostderr=false：不输出到stderr
--log-dir=/var/log/kubernetes:日志的存放目录
--alsologtostderr=false:设置为true表示日志输出到文件也输出到stderr
--v=0:glog的日志级别
--vmodule=gfs=2,test=4：glog基于模块的详细日志级别

4. 常见问题

4.1. Pod状态一直为Pending

kubectl describe <pod_name> --namespace=<NAMESPACE>

查看该POD的事件。

正在下载镜像但拉取不下来（镜像拉取耗时太久）[一般都是该原因]
没有可用的Node可调度
开启了资源配额管理并且当前Pod的目标节点上恰好没有可用的资源

解决方法：

查看该POD所在宿主机与镜像仓库之间的网络是否有问题，可以手动拉取镜像
删除POD实例，让POD调度到别的宿主机上

4.2. Pod创建后不断重启

kubectl get pods中Pod状态一会running，一会不是，且RESTARTS次数不断增加。

一般原因为容器启动命令不是阻塞式命令，导致容器运行后马上退出。

非阻塞式命令：

本身CMD指定的命令就是非阻塞式命令
将服务启动方式设置为后台运行

解决方法：

1、将命令改为阻塞式命令（前台运行），例如：zkServer.sh start-foreground

2、java运行程序的启动脚本将 nohup xxx &的nobup和&去掉，例如：

nohup JAVA_HOME/bin/java JAVA_OPTS -cp $CLASSPATH com.cnc.open.processor.Main &

改为：

JAVA_HOME/bin/java JAVA_OPTS -cp $CLASSPATH com.cnc.open.processor.Main

文章参考《Kubernetes权威指南》

kubernetes问题排查相关推荐

kubernetes 的安装与部署
文章目录一.前置工作与注意事项二.安装 1. 初始准备 2. docker安装 3. k8s安装三.网络插件的安装 1. 常用网络插件 2. 插件安装四.安装dashboard 1.注意事项 ...
windows无法配置此无线连接_Kubernetes 1.18功能详解：OIDC发现、Windows节点支持，还有哪些新特性值得期待？...
Kubernetes 1.18发布,一些对社区产生影响的新特性日渐完善,如 KSA(Kubernetes Service Account) tokens的OIDC发现和对Windows节点的支持.在A ...
K8S常见错误、原因及处理方法
OOMKilled: Pod 的内存使用超出了 resources.limits 中的限制,被强制杀死. CrashLoopBackoff: Pod 进入崩溃-重启循环,重启间隔时间从 10 20 ...
k8s集群DNS无法解析问题的处理过程
搭建k8s集群时DNS无法解析问题的处理过程问题描述在搭建Kubernetes集群过程中,安装了kube-dns插件后,运行一个ubuntu容器,发现容器内无法解析集群外域名,一开始可以解析集群内 ...
docker修改容器映射的端口_解密 Docker 挂载文件，宿主机修改后容器里文件没有修改...
问题使用 Docker Volumes 时,有时需要挂载一个宿主机目录或者文件,提供数据可持续或者容器内部服务配置文件. 使用命令 docker run -it --rm -v /root/test ...
linux中var的作用,linux中var是什么意思
在搭建Kubernetes集群过程中,安装了kube-dns插件后,运行一个ubuntu容器,发现容器内无法解析集群外域名,一开始可以解析集群内域名,一段时间后也无法解析集群内域名. $ nslook ...
Docker 容器退出状态码existed
今天在起容器的时候,发现容器出于existed状态,状态码为130,查找后发现,自己启动的时候使用了非后台启动.当手动 ctrl+C 退出命令行的时候,容器也就退出了.开始的时候,不知道状态码含义,纠 ...
从4方面解释，什么叫云原生应用？
编者注:本文作者吕建伟,作者重点从4个层面全面介绍了云原生的起源和发展,帮助读者学习云原生概念. 从Function到Service 一.从函数说起我是1993年学习电脑的.学习的开发语言有三种:汇 ...
K8S备份、恢复、迁移神器 Velero
前言 Kubernetes 集群备份一直是我们的痛点.虽然可以通过Etcd v3备份与恢复来实现K8S集群备份,但是这种备份很难恢复单个 Namespace. 今天推荐 Velero 工具,它提供以下 ...

kubernetes问题排查