使用G1后报错-CircuitBreakingException: [parent] Data too large

使用G1后报错

Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [<transport_request>]

ES: 7.5.0

今天在优化ES的GC配置的时候又踩了坑，因为之前在7.1上使用过G1，感觉效果还不错，这次的7.5上也就直接升级成了G1，jvm.options的配置是

-Xms24g
-Xmx24g## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly## G1GC Configuration
# NOTE: G1GC is only supported on JDK version 10 or later.
# To use G1GC uncomment the lines below.
10-:-XX:-UseConcMarkSweepGC
10-:-XX:-UseCMSInitiatingOccupancyOnly
10-:-XX:+UseG1GC
10-:-XX:InitiatingHeapOccupancyPercent=75

升级完成之后，集群中的节点在查询的时候有时候会报错

Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [<transport_request>] would be [24714355312/23gb], which is larger than the limit of [24481313587/22.7gb], real usage: [24714353312/23gb], new bytes reserved: [2000/1.9kb], usages [request=0/0b, fielddata=11807/11.5kb, in_flight_requests=2932/2.8kb, accounting=45086480/42.9mb]at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:343) ~[elasticsearch-7.5.0.jar:7.5.0]at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128) ~[elasticsearch-7.5.0.jar:7.5.0]

从这个报错来看是触发了集群的熔断操作导致的，最开始没有注意，后来发现在低流量的时候也很容易触发复现，感觉问题似乎并不简单。使用关键字看到官方有这样的文档

G1 GC were setup to use an InitiatingHeapOccupancyPercent of 75. This
could leave used memory at a very high level for an extended duration,
triggering the real memory circuit breaker even at low activity levels.
The value is a threshold for old generation usage relative to total heap
size and thus it should leave room for the new generation. Default in
G1 is to allow up to 60 percent for new generation and this could mean that the
threshold was effectively at 135% heap usage. GC would still kick in of course and
eventually enough mixed collections would take place such that adaptive adjustment
of IHOP kicks in.The JVM has adaptive setting of the IHOP, but this does not kick in
until it has sampled a few collections. A newly started, relatively
quiet server with primarily new generation activity could thus
experience heap above 95% frequently for a duration.The changes here are two-fold:Use 30% default for IHOP (the JVM default of 45 could still mean
105% heap usage threshold and did not fully ensure not to hit the
circuit breaker with low activity)
Set G1ReservePercent=25. This is used by the adaptive IHOP mechanism,
meaning old/mixed GC should kick in no later than at 75% heap. This
ensures IHOP stays compatible with the real memory circuit breaker also
after being adjusted by adaptive IHOP.

也就是说

10-:-XX:InitiatingHeapOccupancyPercent=75

这个配置是有问题的，这个配置了老年代的初始化内存，但是G1允许新生代使用60%的jvm内存，所以总的内存量可能很容易达到135%,因为这个时候可能并没有占满内存，G1的回收可能不会触发，但是对于ES来说，已经触发了内存熔断机制，所以即使请求量级不大，可能也会频繁报内存熔断的错误，需要注意哦。

修正的方法是

10-:-XX:-UseConcMarkSweepGC
10-:-XX:-UseCMSInitiatingOccupancyOnly
10-:-XX:+UseG1GC
10-:-XX:G1ReservePercent=25
10-:-XX:InitiatingHeapOccupancyPercent=30

增加和修改了配置

10-:-XX:G1ReservePercent=25
10-:-XX:InitiatingHeapOccupancyPercent=30

坑死了，刚重启完毕，又要重启一遍

使用G1后报错-CircuitBreakingException: [parent] Data too large相关推荐

Kibana查询ES报错:[circuit_breaking_exception] [parent] Data too large
一.kibana浏览器报错 {"statusCode":429,"error":"Too Many Requests","mess ...
【es】FATAL [circuit_breaking_exception] [parent] Data too large, data for [＜http_request＞] would be
1.背景本地运行etl集群,一个es,一个head,一个kibana,然后如下上图显示es还初处于正常状态,但是head已经不能访问了然后kibana报错如下 FATAL [circuit_br ...
Elasticsearch-circuit_breaking_exception [parent] Data too large, data for [＜http_request＞]
文章目录异常信息不同版本的配置的差异 ES6.4 ES7.6 Solutions 异常信息 {"root_cause":[{"type":"cir ...
Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data
以上异常原因是field data 的缓存不够用. 修改方法,向Elasticsearch发送如下请求,设置fielddata缓存占用jvm内存的30%或者更小: 之后再发送请求即可成功.
【Elasticsearch】Data too large, data for which is larger than the limit of
1.概述参考:elasticsearch报Data too large异常处理在线上ES集群日志中发现了如下异常,elasticsearch版本为7.3.2 [2021-03-16T21:05:1 ...
elasticsearch线上调优[ Data too large]
项目场景: 最近es的线上经常出现一个内存的异常的问题,我解决了各种的途径都无效.最后再方案一种解决最终办法,方案二能解决部分问题. 问题描述:[[parent] Data too large, da ...
记录一次尝试修复elasticsearch Data too large问题
前言这周三,一边调试api crashloopbackoff的问题,一边准备写promSQL语句,然后开发小伙伴说环境3的es又挂了,表现就是查询数据报错: circuit_breaking_exc ...
解决ES Data too large问题
问题描述: 最近搭建ELK测试系统,运行一段时间后,Kibana无法连接,并且报以下错误: [parent] Data too large, data for [<http_request> ...
[fielddata] Data too large, data for [_id] would be [13181907968/12.2gb]
问题描述使用ES search after查询报如下错误,fielddata缓存熔断异常. org.frameworkset.elasticsearch.ElasticSearchException ...

使用G1后报错-CircuitBreakingException: [parent] Data too large

使用G1后报错-CircuitBreakingException: [parent] Data too large相关推荐

最新文章

热门文章