问题描述

本人最近负责了定时获取Prometheus Metrics并发送到ES做持久化存储的任务。然而在Metrics采集粒度从3分钟变为1小时后（这里使用的是Springboot的定时采集组件Scheduler），报了如下的错误：

java.io.IOException: Connection reset by peerat org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:793) ~[elasticsearch-rest-client-7.4.0.jar!/:7.4.0]at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218) ~[elasticsearch-rest-client-7.4.0.jar!/:7.4.0]at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205) ~[elasticsearch-rest-client-7.4.0.jar!/:7.4.0]at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1454) ~[elasticsearch-rest-high-level-client-7.4.0.jar!/:7.4.0]at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1439) ~[elasticsearch-rest-high-level-client-7.4.0.jar!/:7.4.0]at org.elasticsearch.client.IndicesClient.exists(IndicesClient.java:785) ~[elasticsearch-rest-high-level-client-7.4.0.jar!/:7.4.0]at com.free4inno.scheduler.adapter.service.elasticsearch.GenericEsService.isIndexExists(GenericEsService.java:60) ~[classes!/:0.0.1-SNAPSHOT]at com.free4inno.scheduler.adapter.service.elasticsearch.MetricsToEsService.getTodayIndex(MetricsToEsService.java:56) ~[classes!/:0.0.1-SNAPSHOT]at com.free4inno.scheduler.adapter.service.elasticsearch.MetricsToEsService.insert(MetricsToEsService.java:49) ~[classes!/:0.0.1-SNAPSHOT]at com.free4inno.scheduler.adapter.service.Scheduler.sendMetrics(Scheduler.java:64) [classes!/:0.0.1-SNAPSHOT]at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_261]at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_261]at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_261]at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_261]at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) [spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93) [spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_261]at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_261]at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_261]at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_261]at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_261]at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_261]at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261]
Caused by: java.io.IOException: Connection reset by peerat sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_261]at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_261]at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_261]at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:1.8.0_261]at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:378) ~[?:1.8.0_261]at org.apache.http.impl.nio.reactor.SessionInputBufferImpl.fill(SessionInputBufferImpl.java:231) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.codecs.AbstractMessageParser.fillBuffer(AbstractMessageParser.java:136) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:241) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.4.jar!/:4.1.4]at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar!/:4.1.4]at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.13.jar!/:4.4.13]at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.13.jar!/:4.4.13]... 1 more

更令人费解的是，在采集Metrics发送至ES的过程中，在单数的整点写入ES是正常的，然而一到双数的整点就不正常了，例如0:00开始发送是可以发送到ES的，1:00就发送不到报错，2:00就重新恢复正常了…
后面经过调研，是因为esClient自动设置的KeepAlive时间为-1，也就是持续连接，然而这回受到外界的影响比如Firewall，会将TCP连接单方面断开，从而会导致Connection reset by peer的报错。

解决方案

在设置RestHighLevelClient的时候手动设置一下setKeepAliveStrategy的方法，在这里设置的是3分钟的时间（不要太长）

@Beanpublic RestHighLevelClient restHighLevelClient(@Autowired RestClientBuilder restClientBuilder){return new RestHighLevelClient(restClientBuilder.setHttpClientConfigCallback(requestConfig ->requestConfig.setKeepAliveStrategy((response, context) -> TimeUnit.MINUTES.toMillis(3))));}

或者也可以使用心跳，隔一段时间就去获取一次es的连接状态，以此来保持连接的活跃，个人认为这是下策，这里就没有进行尝试。

https://blog.csdn.net/qq_33999844/article/details/113843845

Elasticsearch出现Connection reset by peer分析

1.异常：

Caused by: java.io.IOException: Connection reset by peer
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:793)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1454)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1424)
2.分析

RestHighLevelClient客户端，使用的是apache httpclient，版本为4.x，keepAlive默认为-1（客户端会一直保持session）；

当服务端因为超时或者其他原因关闭session，客户端仍然认为长连接存在，抛出异常；

注：当ES服务端的keepAlive短于ES客户端的keepAlive，也会导致：服务端已经关闭了连接，客户端继续复用该连接，抛出异常。

3.解决

手动设置KeepAliveStrategy来配置keepAlive，保证客户端keepAlive小于服务端keepAlive，让客户端先于服务端关闭连接
————————————————
版权声明：本文为CSDN博主「夜-NULL(zmc)」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/qq_33999844/article/details/113843845

ES报错：Connection reset by peer 解决经历相关推荐

Connection reset by peer解决方法
当我们用HttpClient请求一个接口的时候,在本地调试都没有问题,但是放到某些服务器上面就会出现Connection reset by peer,(我们的服务器是中控主机),可能一方面使中控主机的 ...
JDBC Connection Fails With SEVERE: IO Error: Connection Reset By Peer解决方法
问题描述: 2021.10.20 研发同事反馈一个关于JDBC Connection错误的问题,具体情况如下: JDBC connection with TCPS from Orcle Enterpr ...
Socket通信客户端报错Connection reset
Socket服务端启动不报错,在启动客户端后控制台出现如下问题 [ Exception in thread "main" java.net.SocketException: Con ...
git报错Connection reset by 140.82.113.4 port 22
git报错如下: Connection reset by 140.82.113.4 port 22 fatal: Could not read from remote repository. Ple ...
python requests 报错 Connection aborted ConnectionResetError RemoteDisconnected 解决方法
错误:requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end close ...
ES ElasticSearch Connection reset by peer问题解决
问题现象程序隔一段时间不操作后,再请求es就会报错Connection reset by peer,之后连续几次操作都正常问题原因报这个错的原因是因为服务端已经关闭了链接,但是客户端还在使用这 ...
ab apr_socket_recv: Connection reset by peer 错误解决
来源:http://blog.csdn.net/largetalk/article/details/7337102 在ubuntu 11.04上的ab(apache-utils)有些问题, 在并发数稍 ...
[Python] ConnectionResetError: [Errno 104] Connection reset by peer
python socket 编程,sftp远程读日志文件,取最后的30行返回.收到ConnectionResetError: [Errno 104] Connection reset by peer报 ...
【es】ES RestHighLevelClient 请求报错：Connection reset by peer
1.概述转载源码解读相似问题:[Elasticsearch]解决Elasticsearch HTTP方式查询报SocketTimeoutException的问题 Connection reset ...

ES报错：Connection reset by peer 解决经历

问题描述

解决方案

Elasticsearch出现Connection reset by peer分析

ES报错：Connection reset by peer 解决经历相关推荐

最新文章

热门文章