Kafka - which is larger than the maximum request size you have configured with the max.request

报错问题

ERROR org.springframework.kafka.support.LoggingProducerListener | Exception thrown when sending a message with key='null' and payload='{123, 34, 97, 112, 105, 82, 101, 115, 112, 111, 110, 115, 101, 34, 58, 123, 34, 109, 115, 103, 34, 5...' to topic resp-persist-topic:
org.apache.kafka.common.errors.RecordTooLargeException: The message is 1784072 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.
org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.cloud.stream.binder.kafka.KafkaMessageChannelBinder$ProducerConfigurationMessageHandler@52afb38]; nested exception is org.springframework.kafka.KafkaException: Send failed; nested exception is org.apache.kafka.common.errors.RecordTooLargeException: The message is 1784072 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.

通俗的话来说，生产者发送端有大小限制，超过了这个限制而已（默认的发送一条消息的大小是 1M，如果不配置，当发送的消息大于 1M 时就会报错）

解决方案

限制调大

# server.properties
#（5M）
message.max.bytes=5242880
#（6M）每个分区试图获取的消息字节数。要大于等于message.max.bytes
replica.fetch.max.bytes=6291456# producer.properties
#（5M）请求的最大大小为字节。要小于 message.max.bytes
max.request.size = 5242880# consumer.properties
# （6M）每个提取请求中为每个主题分区提取的消息字节数。要大于等于message.max.bytes
fetch.message.max.bytes=6291456

Ps：记得重启 Kafka Server 才能生效~

分批发送

这里只讲解思路，具体情况需要看具体业务分析，大概就是自己给发送的数据做切割，比如一个 10M 的数据，不改配置的情况下（默认1M），每次最多发送 1M，那么理论上需要发送 10 次；那么，如何切割也是个学问！有些人会说，如果有数组，那根据数组范围切割，或者说根据一堆对象为一组，以此类推，其实这样都不太好，数据解析复杂度直接上升！！！

这里我推荐一种简单粗暴的方式，其实直接转换成 String，然后截取切切切即可，DDDD~ 其实细想，消费者那边始种要等数据完整了才进行业务处理，所以消费者端维护成本是一样的，那么就对比生产者端看下哪种方案维护成本低即可，我个人认为是先转字符串，然后切割是最快的方式~ 当然这里也会出现其他的情况，比如说中途宕机了咋办，数据还未完整，这种我们这里暂时不讨论，因为属于另一个业务情景要解决的技术方案了！

Kafka - which is larger than the maximum request size you have configured with the max.request相关推荐

【kafka】Found a message larger than the maximum fetch size of this consumer on topic
1.场景1 1.1 概述我消费一下kakfa然后报错信息如下 [root@1 kafka]# bin/kafka-console-consumer.sh --zookeeper 1.xxx:2181 ...
【Netcore】This can either be due to a cycle or if the object depth is larger than the maximum allowed
报错信息 An unhandled exception occurred while processing the request. JsonException: A possible object ...
解决因nginx配置导致文件上传失败的问题 The filed file exceeds its maximum permitted size of 1048576 bytes
解决因nginx配置问题导致文件上传失败的问题 1.问题背景 2.问题截图 3.问题分析 4.问题解决(修改nginx的配置) 5.解决之后的"后遗症"(截图) 6.分析并解决&q ...
SpringBoot上传文件报错The field multiFile exceeds its maximum permitted size of 1048576 bytes.
异常 org.apache.tomcat.util.http.fileupload.impl.FileSizeLimitExceededException: The field multiFile e ...
org.springframework.web.multipart.MaxUploadSizeExceededException: Maximum upload size exceeded; nes
使用springboot上传文件,当文件超过2M报出异常异常显示org.springframework.web.multipart.MaxUploadSizeExceededException: M ...
mysql text 最大长度报错 Row size too large. The maximum row size for the used table type
数据类型长度如下 TINYTEXT 256 bytes TEXT 65,535 bytes ~64kb MEDIUMTEXT 16,777,215 bytes ~16MB LONGTEXT 4, ...
get request uni 参数_uni-app 环境配置，uni.request封装，接口配置，全局配置，接口调用的封装...
1.环境配置 (可参考uni-官网的环境配置) common文件夹下新建config.js let url_config = "" if(process.env.NODE_ENV ...
MySQL【问题记录 01】报错 1709 - Index column size too large. The maximum column size is 767 bytes. 可能是最简单的方法
1.问题重现 CREATE TABLE `xxl_job_registry` (`id` int(11) NOT NULL AUTO_INCREMENT,`registry_group` varcha ...
Request和Response-学习笔记02【请求转发和request共享数据、Request_获取ServletContext、request登录案例】
Java后端学习路线笔记汇总表[黑马程序员] Request和Response-学习笔记01[Request_原理和继承体系.Request_获取请求数据][day01] Request和Resp ...
mycat 启动失败 The specified size exceeds the maximum representable size JVM exited while loading the a
错误信息: [root@server1 bin]# ./mycat console Running Mycat-server... wrapper | --> Wrapper Started ...

Kafka - which is larger than the maximum request size you have configured with the max.request

报错问题

解决方案

Kafka - which is larger than the maximum request size you have configured with the max.request相关推荐

最新文章

热门文章