http: TLS handshake error from xxx.xxx.xxx.xxx:xxxx : read tcp xxx.xxx.xxx.xxx:6443-＞xxx.xxx.xxx.xxx

一、问题描述

二、问题分析

解决方法：

一、问题描述

使用阿里云SLB做 kube-apiserver负载，查看kube-apiserver 日志一直在报 TLS handshake error 错误，100.121.5.128、100.120.70.1两个IP也不知道是从哪里来的，频率在每隔2s执行两次，问题如下：

2022/6/15 11:59:32 I0615 11:59:32.213207       1 log.go:172] http: TLS handshake error from 100.120.70.1:25664: read tcp 172.17.47.140:26443->100.120.70.1:25664: read: connection reset by peer
2022/6/15 11:59:32 I0615 11:59:32.660931       1 log.go:172] http: TLS handshake error from 100.121.5.128:18063: read tcp 172.17.47.140:26443->100.121.5.128:18063: read: connection reset by peer
2022/6/15 11:59:34 I0615 11:59:34.192356       1 log.go:172] http: TLS handshake error from 100.120.70.1:49522: read tcp 172.17.47.140:26443->100.120.70.1:49522: read: connection reset by peer
2022/6/15 11:59:34 I0615 11:59:34.746833       1 log.go:172] http: TLS handshake error from 100.121.5.128:31929: read tcp 172.17.47.140:26443->100.121.5.128:31929: read: connection reset by peer
2022/6/15 11:59:36 I0615 11:59:36.270063       1 log.go:172] http: TLS handshake error from 100.120.70.1:28159: read tcp 172.17.47.140:26443->100.120.70.1:28159: read: connection reset by peer
2022/6/15 11:59:36 I0615 11:59:36.815505       1 log.go:172] http: TLS handshake error from 100.121.5.128:45285: read tcp 172.17.47.140:26443->100.121.5.128:45285: read: connection reset by peer
2022/6/15 11:59:38 I0615 11:59:38.201727       1 log.go:172] http: TLS handshake error from 100.120.70.1:21729: read tcp 172.17.47.140:26443->100.120.70.1:21729: read: connection reset by peer

二、问题分析

1、排查未使用SLB集群无此问题

2、排查两个集群配置相同无差别

3、此IP（100.121.5.128、100.120.70.1）未在集群中与宿主机使用

4、排查阿里SLB健康检查

5、当关闭阿里云SLB健康检查后问题停止，分析阿里云健康检查问题

6、查看阿里云SLB健康检查支持两种方式：TCP协议健康检查通过发送SYN握手报文来检测服务器端口是否存活；HTTP协议健康检查通过发送HEAD/GET请求模拟浏览器的访问行为来检查服务器应用是否健康

7、尝试修改为http健康检查由于kube-apiserver使用https协议，健康检查失败

解决方法：

方法一：停掉健康检查

方法二：使用http协议调用勾选http_4xx、http_5xx

方法三：开启kube-apiserver http端口

# vim /etc/kubernetes/manifests/kube-apiserver.yaml- --enable-admission-plugins=NodeRestriction- --enable-bootstrap-token-auth=true- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.pem- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.pem- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client-key.pem- --etcd-servers=https://172.17.47.140:2379,https://172.17.47.141:2379,https://172.17.47.142:2379- --insecure-port=0- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt

修改：

# 修改为
- --insecure-port=8080
# ip 为不限制，默认为127.0.0.1
- --insecure-bind-address=0.0.0.0

重启服务

systemctl daemon-reload
systemctl restart docker
systemctl restart kubelet

SLB配置

参考：

docs/concepts/aws.md: document TLS handshake errors in kube-apiserver · kinvolk/lokomotive@243459f · GitHub