k8s问题记录与解决

一、问题：error: open /var/lib/kubelet/config.yaml: no such file or directory
解决：关键文件缺失，多发生于没有做 kubeadm init就运行了systemctl start kubelet。要先成功运行kubeadm init

二、kubelet.service has more than one ExecStart= setting, which is only allowed for Type=oneshot services. Refusing.
解决：打开/etc/systemd/system/kubelet.service.d/10-kubeadm.conf 中的配置：
   [root@k8s-master ~]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
   # Note: This dropin only works with kubeadm and kubelet v1.11+
   [Service]
   Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
   Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
   # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
   EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
   # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
   # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
   EnvironmentFile=-/etc/sysconfig/kubelet
   要打开注释此# ExecStart=
   ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

三、journalctl -f -u kubelet （-f是 --follow, -u是过滤出kubelet日志）
centos7 查看日志

四、kubeadm安装的k8s，重新安装k8s-mst
   检查 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 中的配置
   systemctl restart kubelet
   kubeadm reset
   kubeadm init --kubernetes-version=v1.13.4 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap
   其他节点执行
   kubeadm reset
   kubeadm join ...命令

五、kubeadm 生成的token过期后，集群增加节点
参考：https://www.jianshu.com/p/a5e379638577

六、[reset] unmounting mounted directories in "/var/lib/kubelet" 卡主
重启机器，重新执行命令

-------------------20190518 update-------------------

七、master所在虚机重启后，kube-master1 kubelet[34770]: E0419 13:52:09.511348 34770 kubelet.go:2266] node "kube-master1" not found，并发现获取到的ip地址为空，ifconfig命令去查看网卡配置情况，却发现根本没有配置eth0/ens33网卡

解决：依次systemctl stop kubelet、systemctl stop docker、systemctl restart network、systemctl restart docker、systemctl restart kubelet、ifconfig、kubectl get node

-------------------20190614 update-------------------

八、kubelet, k8s-node1 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1b7fb9d83e89dbe2815cc10fb1daf342162cb74da30568c0a59585e1dc9329a4" network for pod "wxapp-redis-1": NetworkPlugin cni failed to set up pod "wxapp-redis-1_pnup" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.1.1/24

解决：到有问题的机器，执行如下命令：

[root@k8s-node1 redis]# cd /var/lib/cni/flannel/
[root@k8s-node1 flannel]# ll
total 32
-rw-------. 1 root root 206 Apr 19 11:30 0727fe1a742f28b9a5d5d3188496bdc0aec220599caf6ce8c28f1b9c8ef1b8d4
-rw-------. 1 root root 206 Apr 19 14:30 13776cebe870d3f58982a123e2e32a4a89780c421db1bc425f13f13756822f81
-rw-------. 1 root root 206 Apr 19 11:30 1f249dd31dae8177a4fa5d3009eec7a36a4dccd8a836975f1d798adf43afda51
-rw-------. 1 root root 206 Apr 19 14:34 317e1ce2f78fddd1b36879be5dff169d27fefd4ca191dcd9d85781ab65cc14d8
-rw-------. 1 root root 187 Jun 14 10:05 5b104d16ea2042bd67f8958b8042fff01de3ba6a69f1a62f4b3ded81955c24bb
-rw-------. 1 root root 206 Apr 23 15:30 5bfe39c5fb73ecb09b7343260b8fc2526bb99b7d00a216ab0c76d87d247f3bc0
-rw-------. 1 root root 187 Jun 14 10:05 85de9489e5a4b5091bc40b5dc216a9f15fdb9aa077a28df32cfef97f8abd0c81
-rw-------. 1 root root 206 Apr 19 11:30 db015f887bbedf2ad7731aaa4a321183594a1d2c9b95398bb25f61ad5b052092
[root@k8s-node1 flannel]# systemctl stop docker
[root@k8s-node1 flannel]# systemctl stop kubelet
[root@k8s-node1 flannel]# systemctl stop kube-proxy
Failed to stop kube-proxy.service: Unit kube-proxy.service not loaded.
[root@k8s-node1 flannel]# rm -rf /var/lib/cni/flannel/ && rm -rf /var/lib/cni/networks/cbr0/ && ip link delete cni0
[root@k8s-node1 flannel]# rm -rf /var/lib/cni/networks/cni0/*
[root@k8s-node1 flannel]# systemctl start docker
[root@k8s-node1 flannel]# systemctl start kubelet

-------------------20190622 update-------------------

九、转（kubernetes --> kube-dns 安装 https://blog.csdn.net/zhuchuangang/article/details/76093887 https://www.cnblogs.com/chimeiwangliang/p/8809280.html）

十、转（kubernetes中网络报错问题排查 http://www.mamicode.com/info-detail-2315259.html）

-------------------20200523 update-------------------

十一、May 23 06:21:59 master dockerd-current[14690]: time="2020-05-23T06:21:59.555312805-04:00" level=error msg="Create container failed with error: oci runtime error: container_linux.go:235: starting container process caused \"process_linux.go:258: applying cgroup configuration for process caused \\\"Cannot set property TasksAccounting, or unknown property.\\\"\"\n"
May 23 06:21:59 master kubelet[21888]: E0523 06:21:59.654065 21888 kubelet.go:2266] node "master" not found
May 23 06:21:59 master kubelet[21888]: E0523 06:21:59.754397 21888 kubelet.go:2266] node "master" not found
May 23 06:21:59 master kubelet[21888]: E0523 06:21:59.855185 21888 kubelet.go:2266] node "master" not found
Docker创建容器报错：Cannot set property TasksAccounting, or unknown property.
最近又新配了一个服务器，想用docker简单的配置一下mysql，没想到创建容器时报错：
Error response from daemon: oci runtime error: container_linux.go:235: starting container process caused “process_linux.go:258: applying cgroup configuration for process caused “Cannot set property TasksAccounting, or unknown property.””
问题原因：主要原因还是centos系统版本兼容性问题，如果将系统做更新升级，即可解决。
执行：yum update后在执行以下操作：

[root@web03 .kube]# yum remove -y kubectl.x86_64 kubeadm.x86_64 kubelet.x86_64
Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager
...
Complete!
[root@web03 .kube]# yum list installed |grep kube
cri-tools.x86_64                      1.13.0-0                       @kubernetes
kubectl.x86_64                        1.13.4-0                       @kubernetes
[root@web03 .kube]# yum install -y kubelet-1.13.4-0.x86_64
Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager
...
Complete!
[root@web03 .kube]# yum install -y kubeadm-1.13.4-0.x86_64
Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager
...
Complete!
[root@web03 .kube]# yum list installed |grep kube
cri-tools.x86_64                      1.13.0-0                       @kubernetes
kubeadm.x86_64                        1.13.4-0                       @kubernetes
kubectl.x86_64                        1.13.4-0                       @kubernetes
kubelet.x86_64                        1.13.4-0                       @kubernetes
kubernetes-cni.x86_64                 0.6.0-0                        @kubernetes

十二、k8s使用kube-router网络插件并监控流量状态
https://www.jianshu.com/p/1a3caecc3b6b
附：kubeadm-kuberouter-all-features.yaml

apiVersion: v1
kind: ConfigMap
metadata:name: kube-router-cfgnamespace: kube-systemlabels:tier: nodek8s-app: kube-router
data:cni-conf.json: |{"cniVersion":"0.3.0","name":"mynet","plugins":[{"name":"kubernetes","type":"bridge","bridge":"kube-bridge","isDefaultGateway":true,"ipam":{"type":"host-local"}}]}
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:labels:k8s-app: kube-routertier: nodename: kube-routernamespace: kube-system
spec:template:metadata:labels:k8s-app: kube-routertier: nodeannotations:scheduler.alpha.kubernetes.io/critical-pod: ''spec:serviceAccountName: kube-routerserviceAccount: kube-routercontainers:- name: kube-routerimage: docker.io/cloudnativelabs/kube-routerimagePullPolicy: IfNotPresentargs:- --run-router=true- --run-firewall=true- --run-service-proxy=true- --kubeconfig=/var/lib/kube-router/kubeconfigenv:- name: NODE_NAMEvalueFrom:fieldRef:fieldPath: spec.nodeName- name: KUBE_ROUTER_CNI_CONF_FILEvalue: /etc/cni/net.d/10-kuberouter.conflistlivenessProbe:httpGet:path: /healthzport: 20244initialDelaySeconds: 10periodSeconds: 3resources:requests:cpu: 250mmemory: 250MisecurityContext:privileged: truevolumeMounts:- name: lib-modulesmountPath: /lib/modulesreadOnly: true- name: cni-conf-dirmountPath: /etc/cni/net.d- name: kubeconfigmountPath: /var/lib/kube-routerreadOnly: trueinitContainers:- name: install-cniimage: busyboximagePullPolicy: Alwayscommand:- /bin/sh- -c- set -e -x;if [ ! -f /etc/cni/net.d/10-kuberouter.conflist ]; thenif [ -f /etc/cni/net.d/*.conf ]; thenrm -f /etc/cni/net.d/*.conf;fi;TMP=/etc/cni/net.d/.tmp-kuberouter-cfg;cp /etc/kube-router/cni-conf.json ${TMP};mv ${TMP} /etc/cni/net.d/10-kuberouter.conflist;fivolumeMounts:- name: cni-conf-dirmountPath: /etc/cni/net.d- name: kube-router-cfgmountPath: /etc/kube-routerhostNetwork: truetolerations:- key: CriticalAddonsOnlyoperator: Exists- effect: NoSchedulekey: node-role.kubernetes.io/masteroperator: Exists- effect: NoSchedulekey: node.kubernetes.io/not-readyoperator: Existsvolumes:- name: lib-moduleshostPath:path: /lib/modules- name: cni-conf-dirhostPath:path: /etc/cni/net.d- name: kube-router-cfgconfigMap:name: kube-router-cfg- name: kubeconfigconfigMap:name: kube-proxyitems:- key: kubeconfig.confpath: kubeconfig
---
apiVersion: v1
kind: ServiceAccount
metadata:name: kube-routernamespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:name: kube-routernamespace: kube-system
rules:- apiGroups:- ""resources:- namespaces- pods- services- nodes- endpointsverbs:- list- get- watch- apiGroups:- "networking.k8s.io"resources:- networkpoliciesverbs:- list- get- watch- apiGroups:- extensionsresources:- networkpoliciesverbs:- get- list- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:name: kube-router
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: kube-router
subjects:
- kind: ServiceAccountname: kube-routernamespace: kube-system

十三、k8s小知识点：如何安装指定版本的kubeadm
https://www.jianshu.com/p/4b22b5d2f69b

-------------------20200523 update-------------------

文献：

1.kubernetes---CentOS7安装kubernetes1.11.2图文完整版

2.安装k8s 1.9.0 实践：问题集锦

k8s问题记录与解决相关推荐

【记录】解决uni-app 用nginx反向代理出现Invalid Host header问题
[记录]解决uni-app 用nginx反向代理出现Invalid Host header问题参考文章: (1)[记录]解决uni-app 用nginx反向代理出现Invalid Host head ...
php curl 不验证ssl,PHP Curl https跳过ssl证书认证报错记录及解决
PHP Curl https跳过ssl证书认证报错记录及解决 function get($url = '', $cookie = '') { $ch = curl_init(); curl_setop ...
C#调用Microsoft.DirectX.DirectSound问题记录及解决
C#调用Microsoft.DirectX.DirectSound问题记录及解决参考文章: (1)C#调用Microsoft.DirectX.DirectSound问题记录及解决 (2)https: ...
K8s入门-K8s节点notReady状态解决
K8s节点notReady状态解决挂掉原因:我想要通过externalIP来发布一个service,同时要删除旧的pod,删除命令执行后,节点就不可用了. 错误操作复现创建externalIP类型 ...
【BUG记录】解决Failure to find (pom/jar) in http://xxx.com was cached in the local repository
[BUG记录]解决Failure to find (pom/jar) in http://xxx.com was cached in the local repository ☠注意:此解决方法仅针对 ...
计算机搜索记录为什么删除不掉,删除电脑上搜索引擎记录的解决方法
现如今,网络的发展给我们的生活带来了很多的便利,让我们的生活与外界紧密联系在一起.但是也给我们带来了不少的困扰,我们的隐私安全得不到保障.特别是我们在利用浏览器搜索东西的记录,几乎是完全曝光的.因此, ...
python使用QQ发邮件成功后发件箱无记录bug解决
在python中使用smtp发送邮件,发现对方有收到,而我自己邮箱的已发送中没有看到发送记录: 解决:QQ邮箱->设置->账户->勾选SMTP送消息保存到服务器->保存更改再 ...
lol 服务器维护看不了比赛记录,LOL无法连接到比赛记录【解决方法】
LOL无法连接到比赛记录原因玩英雄联盟提示无法连接到比赛记录怎么办?其实出现这种问题大多则是因为系统服务器比赛记录出现故障所导致的问题,有时并非是玩家自身的问题,但不排除某些玩家因自身网络问题而导致 ...
K8S 证书过期的解决方法
K8S 证书过期的解决方法 2年前用虚拟机做K8S集群学习.最近打开当时的虚拟机,发现K8S无法启动. The connection to the server xxxx:6443 was refus ...
【vue-seamless-scroll bug问题记录以及解决】
vue-seamless-scroll bug问题记录以及解决 vue-seamless-scroll 的使用方式,请查看网站链接: 组件官方示例. 问题1:无法自动滚动解决方案: ①排查limit ...

k8s问题记录与解决

k8s问题记录与解决相关推荐

最新文章

热门文章