Traefik访问master节点不通的问题定位
- 问题
部署traefik到客户节点的对外访问节点后,发现日志里面报错
类似于
E0122 06:56:04.774977 1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://172.254.0.1:6443/api/v1/endpoints?resourceVersion=0: dial tcp 172.254.0.1:6443: i/o timeout I0122 06:56:04.775358 1 dns.go:174] Waiting for services and endpoints to be ...
172.254.0.1是指向Kubernetes master的地址,跑到出错节点上运行
curl https://172.254.0.1:6443/api/v1/endpoints?resourceVersion=0
发现确实是连接超时,排除证书或者rbac的权限问题后。
- tcpdump分析
发现之前,启动master节点apiserver, advertiese-address都是指到自己的地址,而非vip,修改成vip后重新启动问题依旧
登录节点,通过tcpdump进行分析
tcpdump -i eth0 -nn host master-vip tcpdump -i eth0 -nn host master1 tcpdump -i eth0 -nn host master2 tcpdump -i eth0 -nn host master3
发现不断有请求往master1上发报文,而没有返回,确定是这个问题导致。
查看iptables的报文转发规则,果然发现有些规则
iptables -S -t nat | grep master1-ip
清空iptable也又重新加回去了
- etcd数据
登录etcd server,
export ETCDCTL_API=3 etcdctl --cert=/etc/etcd/ssl/etcd.pem --cacert=/etc/etcd/ssl/ca.pem --key=/etc/etcd/ssl/etcd-key.pem get /registry/services/endpoints/default/kubernetes
终于发现里面写了三条master和vip的记录
del删除后,apiserver立刻只把advertise-address也就是vip重新写入。
traefik应用恢复正常。
查看etcd内容指南
https://yq.aliyun.com/articles/561888
export ETCDCTL_API=3
etcd有目录结构类似linux文件系统,获取所有key看一看:
etcdctl get / --prefix --keys-only
一看就可以大概理解kubenetes的数据结构了,查询命名空间下所有部署的数据:
etcdctl get /registry/deployments/default --prefix --keys-only
etcdctl del /registry/deployments/default/elevated-dragonfly-spinn-front50
删除deployments,pods这可以了,稍微减少一些资源,让kube-apiserver可以正常工作即可,其它资源还可以使用kubectl工具删除
删掉些资源后退出etcd把kube-apiserver的编排文件放回/etc/kubernetes/manifests目录,服务会再次启动,然后再清理重新部署。
etcdctl get / --prefix --keys-only
/registry/apiregistration.k8s.io/apiservices/v1.
/registry/apiregistration.k8s.io/apiservices/v1.apps
/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.autoscaling
/registry/apiregistration.k8s.io/apiservices/v1.batch
/registry/apiregistration.k8s.io/apiservices/v1.networking.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.storage.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.apiextensions.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.apps
/registry/apiregistration.k8s.io/apiservices/v1beta1.authentication.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.batch
/registry/apiregistration.k8s.io/apiservices/v1beta1.certificates.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.events.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.extensions
/registry/apiregistration.k8s.io/apiservices/v1beta1.policy
/registry/apiregistration.k8s.io/apiservices/v1beta1.rbac.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta2.apps
/registry/apiregistration.k8s.io/apiservices/v2beta1.autoscaling
/registry/certificatesigningrequests/node-csr-NMpLG7rhBYiuAyRD739h79x2qhbhWn-UtLPFwJWJU1Y
/registry/certificatesigningrequests/node-csr-v96LRHUfeRD3iAZgHdUYxDaT1kkHvIilDnLZy2GPSew
/registry/clusterrolebindings/cluster-admin
/registry/clusterrolebindings/flannel
/registry/clusterrolebindings/heapster
/registry/clusterrolebindings/kubeadm:kubelet-bootstrap
/registry/clusterrolebindings/kubeadm:node-autoapprove-bootstrap
/registry/clusterrolebindings/kubeadm:node-autoapprove-certificate-rotation
/registry/clusterrolebindings/kubeadm:node-proxier
/registry/clusterrolebindings/kubernetes-dashboard
/registry/clusterrolebindings/system:aws-cloud-provider
/registry/clusterrolebindings/system:basic-user
/registry/clusterrolebindings/system:controller:attachdetach-controller
/registry/clusterrolebindings/system:controller:certificate-controller
/registry/clusterrolebindings/system:controller:clusterrole-aggregation-controller
/registry/clusterrolebindings/system:controller:cronjob-controller
/registry/clusterrolebindings/system:controller:daemon-set-controller
/registry/clusterrolebindings/system:controller:deployment-controller
/registry/clusterrolebindings/system:controller:disruption-controller
/registry/clusterrolebindings/system:controller:endpoint-controller
/registry/clusterrolebindings/system:controller:generic-garbage-collector
/registry/clusterrolebindings/system:controller:horizontal-pod-autoscaler
/registry/clusterrolebindings/system:controller:job-controller
/registry/clusterrolebindings/system:controller:namespace-controller
/registry/clusterrolebindings/system:controller:node-controller
/registry/clusterrolebindings/system:controller:persistent-volume-binder
/registry/clusterrolebindings/system:controller:pod-garbage-collector
/registry/clusterrolebindings/system:controller:replicaset-controller
/registry/clusterrolebindings/system:controller:replication-controller
/registry/clusterrolebindings/system:controller:resourcequota-controller
/registry/clusterrolebindings/system:controller:route-controller
/registry/clusterrolebindings/system:controller:service-account-controller
/registry/clusterrolebindings/system:controller:service-controller
/registry/clusterrolebindings/system:controller:statefulset-controller
/registry/clusterrolebindings/system:controller:ttl-controller
/registry/clusterrolebindings/system:coredns
/registry/clusterrolebindings/system:discovery
/registry/clusterrolebindings/system:kube-controller-manager
/registry/clusterrolebindings/system:kube-dns
/registry/clusterrolebindings/system:kube-scheduler
/registry/clusterrolebindings/system:node
/registry/clusterrolebindings/system:node-proxier
/registry/clusterroles/admin
/registry/clusterroles/cluster-admin
/registry/clusterroles/edit
/registry/clusterroles/flannel
/registry/clusterroles/system:aggregate-to-admin
/registry/clusterroles/system:aggregate-to-edit
/registry/clusterroles/system:aggregate-to-view
/registry/clusterroles/system:auth-delegator
/registry/clusterroles/system:aws-cloud-provider
/registry/clusterroles/system:basic-user
/registry/clusterroles/system:certificates.k8s.io:certificatesigningrequests:nodeclient
/registry/clusterroles/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
/registry/clusterroles/system:controller:attachdetach-controller
/registry/clusterroles/system:controller:certificate-controller
/registry/clusterroles/system:controller:clusterrole-aggregation-controller
/registry/clusterroles/system:controller:cronjob-controller
/registry/clusterroles/system:controller:daemon-set-controller
/registry/clusterroles/system:controller:deployment-controller
/registry/clusterroles/system:controller:disruption-controller
/registry/clusterroles/system:controller:endpoint-controller
/registry/clusterroles/system:controller:generic-garbage-collector
/registry/clusterroles/system:controller:horizontal-pod-autoscaler
/registry/clusterroles/system:controller:job-controller
/registry/clusterroles/system:controller:namespace-controller
/registry/clusterroles/system:controller:node-controller
/registry/clusterroles/system:controller:persistent-volume-binder
/registry/clusterroles/system:controller:pod-garbage-collector
/registry/clusterroles/system:controller:replicaset-controller
/registry/clusterroles/system:controller:replication-controller
/registry/clusterroles/system:controller:resourcequota-controller
/registry/clusterroles/system:controller:route-controller
/registry/clusterroles/system:controller:service-account-controller
/registry/clusterroles/system:controller:service-controller
/registry/clusterroles/system:controller:statefulset-controller
/registry/clusterroles/system:controller:ttl-controller
/registry/clusterroles/system:coredns
/registry/clusterroles/system:discovery
/registry/clusterroles/system:heapster
/registry/clusterroles/system:kube-aggregator
/registry/clusterroles/system:kube-controller-manager
/registry/clusterroles/system:kube-dns
/registry/clusterroles/system:kube-scheduler
/registry/clusterroles/system:node
/registry/clusterroles/system:node-bootstrapper
/registry/clusterroles/system:node-problem-detector
/registry/clusterroles/system:node-proxier
/registry/clusterroles/system:persistent-volume-provisioner
/registry/clusterroles/view
/registry/configmaps/kube-public/cluster-info
/registry/configmaps/kube-system/coredns
/registry/configmaps/kube-system/extension-apiserver-authentication
/registry/configmaps/kube-system/kube-flannel-cfg
/registry/configmaps/kube-system/kube-proxy
/registry/configmaps/kube-system/kubeadm-config
/registry/configmaps/kube-system/kubernetes-dashboard-settings
/registry/controllerrevisions/kube-system/kube-flannel-ds-dd8484577
/registry/controllerrevisions/kube-system/kube-proxy-95cfb667c
/registry/cronjobs/default/hello
/registry/daemonsets/kube-system/kube-flannel-ds
/registry/daemonsets/kube-system/kube-proxy
/registry/deployments/default/flower
/registry/deployments/default/pc-saveapi
/registry/deployments/default/pxxxm-django
/registry/deployments/default/pxxxm-redis
/registry/deployments/default/pxxxm-tengine-proxy
/registry/deployments/default/pxxxm-tengine-static
/registry/deployments/default/pxxxmlogapi-nj
/registry/deployments/kube-system/coredns
/registry/deployments/kube-system/heapster
/registry/deployments/kube-system/kubernetes-dashboard
/registry/deployments/kube-system/monitoring-grafana
/registry/deployments/kube-system/monitoring-influxdb
/registry/events/default/flower-554b58b968-n48sj.151de36ccc190d41
/registry/events/default/flower-554b58b968-n48sj.151de36e0d128ca9
/registry/events/default/hello.151e1bcd521ee391
/registry/jobs/default/hello-1521622920
/registry/jobs/default/hello-1521622980
/registry/jobs/default/hello-1521623040
/registry/minions/cnsz131381
/registry/minions/cnsz131382
/registry/minions/cnsz131383
/registry/namespaces/default
/registry/namespaces/kube-public
/registry/namespaces/kube-system
/registry/pods/default/flower-554b58b968-n48sj
/registry/pods/default/hello-1521622920-5z4bl
/registry/pods/default/hello-1521622980-4qcrd
/registry/pods/default/hello-1521623040-szf4f
/registry/pods/default/pc-saveapi-58cf87b4f5-5x2d8
/registry/pods/default/pxxxm-django-8466f46d84-4mjtf
/registry/pods/default/pxxxm-django-8466f46d84-pnqzp
/registry/pods/default/pxxxm-django-8466f46d84-x5d8z
/registry/pods/default/pxxxm-redis-767c5d5966-l88qj
/registry/pods/default/pxxxm-tengine-proxy-75d874d898-8275h
/registry/pods/default/pxxxm-tengine-static-7d7bb5d5b6-7568z
/registry/pods/default/pxxxmlogapi-nj-687cc89b96-ps4np
/registry/pods/kube-system/coredns-65dcdb4cf-hgmdv
/registry/pods/kube-system/etcd-cnsz131383
/registry/pods/kube-system/heapster-5996df699d-d8798
/registry/pods/kube-system/kube-apiserver-cnsz131383
/registry/pods/kube-system/kube-controller-manager-cnsz131383
/registry/pods/kube-system/kube-flannel-ds-gr8jp
/registry/pods/kube-system/kube-flannel-ds-p4tpq
/registry/pods/kube-system/kube-flannel-ds-z7s96
/registry/pods/kube-system/kube-proxy-cgjhk
/registry/pods/kube-system/kube-proxy-f7snz
/registry/pods/kube-system/kube-proxy-l98gl
/registry/pods/kube-system/kube-scheduler-cnsz131383
/registry/pods/kube-system/kubernetes-dashboard-5bd6f767c7-vr6sg
/registry/pods/kube-system/monitoring-grafana-6bf544559d-cd6d4
/registry/pods/kube-system/monitoring-influxdb-865d95895b-n9sls
/registry/ranges/serviceips
/registry/ranges/servicenodeports
/registry/replicasets/default/flower-554b58b968
/registry/replicasets/default/pc-saveapi-58cf87b4f5
/registry/replicasets/default/pxxxm-django-8466f46d84
/registry/replicasets/default/pxxxm-redis-767c5d5966
/registry/replicasets/default/pxxxm-tengine-proxy-75d874d898
/registry/replicasets/default/pxxxm-tengine-static-7d7bb5d5b6
/registry/replicasets/default/pxxxmlogapi-nj-687cc89b96
/registry/replicasets/kube-system/coredns-65dcdb4cf
/registry/replicasets/kube-system/heapster-5996df699d
/registry/replicasets/kube-system/kubernetes-dashboard-5bd6f767c7
/registry/replicasets/kube-system/monitoring-grafana-6bf544559d
/registry/replicasets/kube-system/monitoring-influxdb-865d95895b
/registry/rolebindings/kube-public/kubeadm:bootstrap-signer-clusterinfo
/registry/rolebindings/kube-public/system:controller:bootstrap-signer
/registry/rolebindings/kube-system/kubernetes-dashboard-minimal
/registry/rolebindings/kube-system/system::leader-locking-kube-controller-manager
/registry/rolebindings/kube-system/system::leader-locking-kube-scheduler
/registry/rolebindings/kube-system/system:controller:bootstrap-signer
/registry/rolebindings/kube-system/system:controller:cloud-provider
/registry/rolebindings/kube-system/system:controller:token-cleaner
/registry/roles/kube-public/kubeadm:bootstrap-signer-clusterinfo
/registry/roles/kube-public/system:controller:bootstrap-signer
/registry/roles/kube-system/extension-apiserver-authentication-reader
/registry/roles/kube-system/kubernetes-dashboard-minimal
/registry/roles/kube-system/system::leader-locking-kube-controller-manager
/registry/roles/kube-system/system::leader-locking-kube-scheduler
/registry/roles/kube-system/system:controller:bootstrap-signer
/registry/roles/kube-system/system:controller:cloud-provider
/registry/roles/kube-system/system:controller:token-cleaner
/registry/secrets/default/default-token-hfj82
/registry/secrets/kube-public/default-token-wpbfs
/registry/secrets/kube-system/attachdetach-controller-token-7jsks
/registry/secrets/kube-system/bootstrap-signer-token-xs87g
/registry/secrets/kube-system/certificate-controller-token-x4swr
/registry/secrets/kube-system/clusterrole-aggregation-controller-token-z7w85
/registry/secrets/kube-system/coredns-token-tbbbw
/registry/secrets/kube-system/cronjob-controller-token-p44bn
/registry/secrets/kube-system/daemon-set-controller-token-4klc2
/registry/secrets/kube-system/default-token-g9lcp
/registry/secrets/kube-system/deployment-controller-token-gfcdt
/registry/secrets/kube-system/disruption-controller-token-vs4dz
/registry/secrets/kube-system/endpoint-controller-token-92kn2
/registry/secrets/kube-system/flannel-token-wvgjq
/registry/secrets/kube-system/generic-garbage-collector-token-pcpnq
/registry/secrets/kube-system/heapster-token-6szhz
/registry/secrets/kube-system/horizontal-pod-autoscaler-token-xqcl5
/registry/secrets/kube-system/job-controller-token-422zr
/registry/secrets/kube-system/kube-proxy-token-k5rfq
/registry/secrets/kube-system/kubernetes-dashboard-certs
/registry/secrets/kube-system/kubernetes-dashboard-key-holder
/registry/secrets/kube-system/kubernetes-dashboard-token-kpkw7
/registry/secrets/kube-system/namespace-controller-token-r57ss
/registry/secrets/kube-system/node-controller-token-5f7sm
/registry/secrets/kube-system/persistent-volume-binder-token-7rnt2
/registry/secrets/kube-system/pod-garbage-collector-token-2txwf
/registry/secrets/kube-system/replicaset-controller-token-zh9xq
/registry/secrets/kube-system/replication-controller-token-q69jz
/registry/secrets/kube-system/resourcequota-controller-token-92hn4
/registry/secrets/kube-system/service-account-controller-token-vvw5d
/registry/secrets/kube-system/service-controller-token-lslnf
/registry/secrets/kube-system/statefulset-controller-token-wdqjt
/registry/secrets/kube-system/token-cleaner-token-9zzpv
/registry/secrets/kube-system/ttl-controller-token-tc7pl
/registry/serviceaccounts/default/default
/registry/serviceaccounts/kube-public/default
/registry/serviceaccounts/kube-system/attachdetach-controller
/registry/serviceaccounts/kube-system/bootstrap-signer
/registry/serviceaccounts/kube-system/certificate-controller
/registry/serviceaccounts/kube-system/clusterrole-aggregation-controller
/registry/serviceaccounts/kube-system/coredns
/registry/serviceaccounts/kube-system/cronjob-controller
/registry/serviceaccounts/kube-system/daemon-set-controller
/registry/serviceaccounts/kube-system/default
/registry/serviceaccounts/kube-system/deployment-controller
/registry/serviceaccounts/kube-system/disruption-controller
/registry/serviceaccounts/kube-system/endpoint-controller
/registry/serviceaccounts/kube-system/flannel
/registry/serviceaccounts/kube-system/generic-garbage-collector
/registry/serviceaccounts/kube-system/heapster
/registry/serviceaccounts/kube-system/horizontal-pod-autoscaler
/registry/serviceaccounts/kube-system/job-controller
/registry/serviceaccounts/kube-system/kube-proxy
/registry/serviceaccounts/kube-system/kubernetes-dashboard
/registry/serviceaccounts/kube-system/namespace-controller
/registry/serviceaccounts/kube-system/node-controller
/registry/serviceaccounts/kube-system/persistent-volume-binder
/registry/serviceaccounts/kube-system/pod-garbage-collector
/registry/serviceaccounts/kube-system/replicaset-controller
/registry/serviceaccounts/kube-system/replication-controller
/registry/serviceaccounts/kube-system/resourcequota-controller
/registry/serviceaccounts/kube-system/service-account-controller
/registry/serviceaccounts/kube-system/service-controller
/registry/serviceaccounts/kube-system/statefulset-controller
/registry/serviceaccounts/kube-system/token-cleaner
/registry/serviceaccounts/kube-system/ttl-controller
/registry/services/endpoints/default/flower
/registry/services/endpoints/default/kubernetes
/registry/services/endpoints/default/pc-saveapi
/registry/services/endpoints/default/pxxxm-django
/registry/services/endpoints/default/pxxxm-redis
/registry/services/endpoints/default/pxxxm-tengine-proxy
/registry/services/endpoints/default/pxxxm-tengine-static
/registry/services/endpoints/default/pxxxmlogapi-nj
/registry/services/endpoints/kube-system/heapster
/registry/services/endpoints/kube-system/kube-controller-manager
/registry/services/endpoints/kube-system/kube-dns
/registry/services/endpoints/kube-system/kube-scheduler
/registry/services/endpoints/kube-system/kubernetes-dashboard
/registry/services/endpoints/kube-system/monitoring-grafana
/registry/services/endpoints/kube-system/monitoring-influxdb
/registry/services/specs/default/flower
/registry/services/specs/default/kubernetes
/registry/services/specs/default/pc-saveapi
/registry/services/specs/default/pxxxm-django
/registry/services/specs/default/pxxxm-redis
/registry/services/specs/default/pxxxm-tengine-proxy
/registry/services/specs/default/pxxxm-tengine-static
/registry/services/specs/default/pxxxmlogapi-nj
/registry/services/specs/kube-system/heapster
/registry/services/specs/kube-system/kube-dns
/registry/services/specs/kube-system/kubernetes-dashboard
/registry/services/specs/kube-system/monitoring-grafana
/registry/services/specs/kube-system/monitoring-influxdb
转载于:https://www.cnblogs.com/ericnie/p/8778082.html
Traefik访问master节点不通的问题定位相关推荐
- K8S——单master节点和基于单master节点的双master节点二进制部署(本机实验,防止卡顿,所以多master就不做3台了)
K8S--单master节点和基于单master节点的双master节点二进制部署 一.准备 二.ETCD集群 1.master节点 2.node节点 三.Flannel网络部署 四.测试容器间互通 ...
- master节点重置后添加node报错_超强教程!在树莓派上构建多节点K8S集群!
在很长一段时间里,我对于在树莓派上搭建Kubernetes集群极为感兴趣.在网络上找到一些教程并且跟着实操,我已经能够将Kubernetes安装在树莓派上,并在三个Pi集群中工作.然而,在master ...
- 部署keepalive+lvs多master节点高可用
最低配置2核 2G 192.168.1.11 master1 192.168.1.12 master2 192.168.1.13 master3 192.168.1.20 node1 一.初始化实验环 ...
- k8s1.18多master节点高可用集群安装-超详细中文官方文档
kubernetes安装系列文章 kubernetes1.17.3安装-超详细的安装步骤 安装kubernetes1.17.3多master节点的高可用集群 k8s1.18单master节点高可用集群 ...
- 搭建K8s集群(kubeadm方式)-部署master节点
安装Docker/kubeadm/kubelet 所有节点安装Docker/kubeadm/kubelet ,Kubernetes默认CRI(容器运行时)为Docker,因此先安装Docker 安装D ...
- k8s ready 不调度_【零基础学云计算】k8s部署---master节点组件部署(三)
kube-APIserver组件介绍 kube-APIserver提供了k8s各类资源对象(pod,RC,Service等)的增删改查及watch等HTTP Rest接口,是整个系统的数据总线和数据中 ...
- kubesphere master节点执行kubectl get po Unable to connect to the server: Forbidden
在kubesphere master节点上执行kubectl get po -n xx报出Unable to connect to the server: Forbidden的错误信息,但是集群整体运 ...
- 如何优雅的修改 Kubernetes Master 节点 IP?可没你想象中那么简单!
公众号关注 「奇妙的 Linux 世界」 设为「星标」,每天带你玩转 Linux ! 昨天网络环境出了点问题,本地的虚拟机搭建的 Kubernetes 环境没有固定 IP,结果节点 IP 变了,当然最 ...
- kubeadm集群化部署多master节点(生产环境适用)
一.背景介绍. k8s通过master集中式管理worknode的容器编排系统,而在生产环境为了维护高可用性,master的地位起到举无轻重的作用.一旦master节点失守,则会导致整个集群服务不可用 ...
最新文章
- java tf值搜索_搜索引擎优化 TF_IDF之Java实现
- 用友ERP服务器的连接
- python分箱分类代码_Python实现变量分箱及应用
- MVC的WebApi中开启Session会话支持
- 虚拟机usb无线上网同时主机本地上网或无线无线上网
- 机器学习全面教程无监督学习系列
- ASP.NET里的路径的使用-预备篇
- 50道经典MySQL练习题(含解答)
- 偏差-方差分解、损失函数和正则化
- 实战1--航空公司客户价值分析(画图篇)
- 牛腩新闻发布--本地超链接打不开
- 双击 文字 出现 文本框 的方法
- ear的英语怎么念_ears用英语怎么读
- 记录element-ui自定义表单验证上传身份证正反面
- vdisk 的缓存系统设计
- 安装mysql中遇到的问题
- 怎么用python制作超级玛丽_Python实现超级玛丽游戏系列教程01玛丽登场
- 微软安全软件_三星与微软合作 共同开发智能手机对讲机功能
- ansible完全攻略
- 《软件安装与使用教程》— Windows操作系统快速安装MATLAB最新最全教程(保姆级教程以MATLAB2019A为例)