kafka报org.apache.kafka.common.errors.RecordTooLargeException

2019独角兽企业重金招聘Python工程师标准>>>

kakfa报错如下：

java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.RecordTooLargeException: The message is 12792083 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.

原因是发送的消息过大，大于默认配置。其源码如下：

ProducerConfig.java

 .define(MAX_REQUEST_SIZE_CONFIG,Type.INT,1 * 1024 * 1024,atLeast(0),Importance.MEDIUM,MAX_REQUEST_SIZE_DOC)

可以看到默认是1M，只需要在配置kafka连接时，加入配置max.request.size即可，如下：

properties.put("bootstrap.servers", "172.16.40.4:9092");
properties.put("acks", "1");
properties.put("retries", 0);
properties.put("batch.size", 16384);
properties.put("linger.ms", 1);
properties.put("max.request.size", 12695150);
properties.put("buffer.memory", 33554432);
properties.put("key.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");
properties.put("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");

但是需要注意的是，在这里配置的值应该小于服务端配置的最大值，否则报如下错误

org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept.

如果要修改服务端配置，则需要修改两个地方，首先是server.properties，加入

message.max.bytes=12695150

然后是producer.properties，加入

max.request.size=12695150

同时，消费端也要配置属性max.partition.fetch.bytes以接收大数据。

转载于:https://my.oschina.net/shyloveliyi/blog/1620012

kafka报org.apache.kafka.common.errors.RecordTooLargeException相关推荐

【记一次kafka报org.apache.kafka.clients.consumer.CommitFailedException异常处理】
项目场景: 项目中,使用到了kafka作为消息中间件,项目作为消费端,消费消息并进行业务处理问题描述在实际应用的过程中,发现偶尔但是一直存在的,有消费数据报:org.apache.kafka.cl ...
【Flink实战系列】Lorg/apache/flink/kafka/shaded/org/apache/kafka/clients/consumer/ConsumerRecord；)Ljava/
java.lang.AbstractMethodError: Method flink/stream/deserialization/PoJoDeserializationSchema.deseria ...
kafka 报错：kafka server：Message was too large ，server rejected it to avoid allocation
kafka报错信息: kafka server:Message was too large ,server rejected it to avoid allocation kafka版本: kafk ...
flink报错ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer
文章目录问题描述报错信息问题解决方式一: 方式二: 问题描述代码如下: public class Sink_KafkaSink_1{public static void main(Strin ...
kafka --org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootst
报错内容: Exception in thread "main" org.apache.kafka.common.KafkaException: Failed to constru ...
kafka报错The request included a message larger than the max message size the server will accept
org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than t ...
Flink读取Kafka报错：KafkaException ByteArrayDeserializer is not an instance Deserializer
1.视界 2.背景做flink读取kafka报错 org.apache.kafka.common.KafkaException: Failed to construct kafka consumer ...
Spring Apache Kafka教程
在本SpringApache Kafka课程中,我们将学习如何在Spring Boot项目中开始使用Apache Kafka,并开始生成和使用我们所选主题的消息. 除了一个简单的项目外,我们还将深入探 ...
2.24. Spring boot with Apache Kafka
Spring boot 1.5.1 2.24.1. 安装 kafka 一下安装仅仅适合开发环境,生产环境请使用这个脚本安装 https://github.com/oscm/shell/tree/mas ...
Apache Kafka Consumer 消费者集
1.目标在我们的上一篇文章中,我们讨论了Kafka Producer.今天,我们将讨论Kafka Consumer.首先,我们将看到什么是Kafka Consumer和Kafka Consumer的 ...

kafka报org.apache.kafka.common.errors.RecordTooLargeException

kafka报org.apache.kafka.common.errors.RecordTooLargeException相关推荐

最新文章

热门文章