Expiring XXX record(s) for XXX:120015 ms has passed since batch creation

问题背景：dws曝光人+场模型聚合压测，憋量20亿左右数据；
问题发生现象：flink job启动后，频繁发生checkpoint失败，并且checkpoint失败原因：Failure reason: Checkpoint was declined.
问题现场日志：

org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 8 for operator aggregate -> Sink: exp sink (86/160). Failure reason: Checkpoint was declined.at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:434) ...
Caused by: org.apache.flink.util.SerializedThrowable: Failed to send data to Kafka: Expiring 2483 record(s) for 【topic_name】-85:120015 ms has passed since batch creation ...
Caused by: org.apache.flink.util.SerializedThrowable: Expiring 2483 record(s) for 【topic_name】-85:120015 ms has passed since batch creation ...
org.apache.flink.runtime.jobmaster.JobMaster - Trying to recover from a global failure.
org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold.

问题发生原因描述：
问题的根本原因是kafka消息发送是批量发送，ProducerRecord会先存储到本地buffer，消息存储在这个buffer里的时长是有限制的【request.timeout.ms】，因此在消息量级比较大，存储在buffer里的消息，超过了request.timeout.ms这个设置时长，就会报上述Expiring XXX record(s) for XXX:120015 ms has passed since batch creation错误；而与此同时，我们开启了端到端的精准一次特性即事务，此时checkpoint与消息的pre commit绑定，pre commit 失败，导致checkpoint的失败，任务重启，大量消息积压；
问题解决方案：
a)调整 request.timeout.ms 这个参数去满足需求，让消息在buffer里待更长的时间；
b)我们公司会给与每个生产者限速，可以提升生产者的速度，这样本地缓存的消息就不会产生积压；
checkpoint失败现场截图，表现为某一个或者多个并行度checkpoint失败：

Expiring XXX record(s) for XXX:120015 ms has passed since batch creation相关推荐

Expiring 1 record(s) for canalEtl_hn-0: 30017 ms has passed since batch creation plus linger time
一问题描述 canal instance报错: Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common ...
kafka record(s) for xxxxxx: 30043 ms has passed since last append异常
在最近使用kafka过程中,有一部分的客户使用平台时会偶发出现以下错误刚开始网上找到的答案是对应主题所在分区对应的宿主机处理出现异常导致,故而导致该异常,所以出问题时取查询时并没有发现kafka s ...
【Kafka】Failed to send data to Kafka: Expiring 30 record(s) for xxx 732453 ms has passed since last a
文章目录 1.美图 2.背景 2.尝试方案1 3.尝试解决2 本文为博主九师兄(QQ:541711153 欢迎来探讨技术)原创文章,未经允许博主不允许转载. 1.美图问题与 [Flink]Flink ...
Expiring 1 record(s) for 2:xxx ms has passed since batch问题研究
本文主要对该错误相关文章进行整理,供自己以及各位参考,如有问题请及时指出,谢谢. 报错大意为:生产发送批次已经创建,但是已经过去120000ms,仍然没有发送,消息过期. [20211015更新](@ ...
【Flink】Flink 写入 kafka 报错 Failed to send data to Kafka: Expiring 4 record(s) for 20001 ms has passed
文章目录 1.场景1 1.1 概述 1.2 百度 1.3 同样问题 2.场景再现本文为博主九师兄(QQ:541711153 欢迎来探讨技术)原创文章,未经允许博主不允许转载. 1.场景1 1.1 概 ...
[2012-05-10 09:52:11 - xxx] Failed to install xxx.apk on device '100046a35a59': timeout
昨天用eclipse开发Android程序时,忽然莫名其妙报这个错,之前还好好的.忽然就不行了. [2012-05-10 09:52:11 - xxx] Failed to install xxx.a ...
TFS2010映射工作区问题路径 XXX 已在工作区 XXX;XXX 中映射
路径 XXX 已在工作区 XXX;XXX 中映射原因:之前用别的用户登录工作区并映射到该路径. 解决办法:再用之前的用户登录工作区取消映射即可. 备忘:在文件 C:\Documents and Se ...
fatal: Path ‘XXX‘ is in submodule ‘XXX‘错误（path is in submodule）
由于需求的原因,最近在修改Netron这个机器学习可视化网络的开源代码,并重新编译,但是在修改成功之后上传到本人的git仓库的时候却出现了 fatal: Path 'XXX' is in submod ...
kafka报错：creating broker listeners from xxx unable to parse xxx:9092 to a broker endpoint
1.美图 2.背景 kafka报错:creating broker listeners from xxx unable to parse xxx:9092 to a broker endpoint 具 ...

Expiring XXX record(s) for XXX:120015 ms has passed since batch creation

Expiring XXX record(s) for XXX:120015 ms has passed since batch creation

Expiring XXX record(s) for XXX:120015 ms has passed since batch creation相关推荐

最新文章

热门文章