关于Netty的ByteBuff内存泄漏问题

之前做的东华车管数据采集平台总是发生数据丢失的情况，虽然不频繁但是还是要关注一下原因，于是今天提高了Netty的Log级别，打算查找一下问题出在哪了，提高级别代码：

ServerBootstrap b =new ServerBootstrap();
b.group(bossGroup,workerGroup).channel(NioServerSocketChannel.class).option(ChannelOption.SO_BACKLOG, 2048).handler(new LoggingHandler(LogLevel.DEBUG)).childHandler(new ChildChannelHandler());

将Loglevel设置成DEBUG模式就OK了。
于是开始安心的观察日志：

2017-01-19 10:04:46  [ nioEventLoopGroup-1-0:1625429 ] - [ INFO ]  消息主体:60160308049620860021010707190117020453395443491162627407087d081f00002e37008801008c00f9
2017-01-19 10:04:49  [ nioEventLoopGroup-1-0:1628830 ] - [ ERROR ]  LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dio.netty.leakDetectionLevel=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information.
2017-01-19 10:04:49  [ nioEventLoopGroup-1-0:1628845 ] - [ INFO ]  入缓存队列操作结果:9
2017-01-19 10:04:49  [ nioEventLoopGroup-1-0:1628845 ] - [ INFO ]  消息主体:601603080496208600210107071901170204573954434611626262170f88091f00002e37008801008c00fa
2017-01-19 10:04:53  [ nioEventLoopGroup-1-0:1632839 ] - [ INFO ]  入缓存队列操作结果:9
2017-01-19 10:04:53  [ nioEventLoopGroup-1-0:1632839 ] - [ INFO ]  消息主体:60160308049620860021010707190117020501395443581162624817108a091f00002e37008801008c00fb
2017-01-19 10:04:55  [ nioEventLoopGroup-1-0:1634196 ] - [ INFO ]  入缓存队列操作结果:9
2017-01-19 10:04:55  [ nioEventLoopGroup-1-0:1634196 ] - [ INFO ]  消息主体:601603080496208600210107071901170205023954436011626244571288091f00002e37008801008c00fc
2017-01-19 10:04:56  [ nioEventLoopGroup-1-0:1635288 ] - [ INFO ]  入缓存队列操作结果:9
2017-01-19 10:04:56  [ nioEventLoopGroup-1-0:1635288 ] - [ INFO ]  消息主体:60160308049620860021010707190117020503395443651162624107118a091f00002e37008801008c00fd
2017-01-19 10:04:57  [ nioEventLoopGroup-1-0:1636443 ] - [ INFO ]  入缓存队列操作结果:9
2017-01-19 10:04:57  [ nioEventLoopGroup-1-0:1636443 ] - [ INFO ]  消息主体:601603080496208600210107071901170205053954437111626234671088091f00002e37008801008c00fe

注意这句话：

LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dio.netty.leakDetectionLevel=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information.

通过这句话我们可以得知，只要加入

ResourceLeakDetector.setLevel(ResourceLeakDetector.Level.ADVANCED);

将警告级别设置成Advaced即可查到更详细的泄漏信息，之后再度查看日志：

2017-01-19 10:35:59  [ nioEventLoopGroup-1-0:665092 ] - [ ERROR ]  LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 5
#5:io.netty.buffer.AdvancedLeakAwareByteBuf.readBytes(AdvancedLeakAwareByteBuf.java:435)com.dhcc.ObdServer.ObdServerHandler.channelRead(ObdServerHandler.java:31)io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:243)io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:956)io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:127)io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:514)io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:471)io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:385)io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:351)io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)io.netty.util.internal.chmv8.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1412)io.netty.util.internal.chmv8.ForkJoinTask.doExec(ForkJoinTask.java:280)io.netty.util.internal.chmv8.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:877)io.netty.util.internal.chmv8.ForkJoinPool.scan(ForkJoinPool.java:1706)io.netty.util.internal.chmv8.ForkJoinPool.runWorker(ForkJoinPool.java:1661)io.netty.util.internal.chmv8.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:126)
#4:Hint: 'ObdServerHandler#0' will handle the message from this point.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:387)io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:243)io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:956)io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:127)io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:514)io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:471)io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:385)io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:351)io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)io.netty.util.internal.chmv8.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1412)io.netty.util.internal.chmv8.ForkJoinTask.doExec(ForkJoinTask.java:280)io.netty.util.internal.chmv8.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:877)io.netty.util.internal.chmv8.ForkJoinPool.scan(ForkJoinPool.java:1706)io.netty.util.internal.chmv8.ForkJoinPool.runWorker(ForkJoinPool.java:1661)io.netty.util.internal.chmv8.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:126)
#3:io.netty.buffer.AdvancedLeakAwareByteBuf.release(AdvancedLeakAwareByteBuf.java:721)io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:237)io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:956)io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:127)io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:514)io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:471)io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:385)io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:351)io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)io.netty.util.internal.chmv8.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1412)io.netty.util.internal.chmv8.ForkJoinTask.doExec(ForkJoinTask.java:280)io.netty.util.internal.chmv8.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:877)io.netty.util.internal.chmv8.ForkJoinPool.scan(ForkJoinPool.java:1706)io.netty.util.internal.chmv8.ForkJoinPool.runWorker(ForkJoinPool.java:1661)io.netty.util.internal.chmv8.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:126)
#2:io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:693)io.netty.handler.codec.DelimiterBasedFrameDecoder.decode(DelimiterBasedFrameDecoder.java:277)io.netty.handler.codec.DelimiterBasedFrameDecoder.decode(DelimiterBasedFrameDecoder.java:216)io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:316)io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:230)io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:956)io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:127)io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:514)io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:471)io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:385)io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:351)io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)io.netty.util.internal.chmv8.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1412)io.netty.util.internal.chmv8.ForkJoinTask.doExec(ForkJoinTask.java:280)io.netty.util.internal.chmv8.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:877)io.netty.util.internal.chmv8.ForkJoinPool.scan(ForkJoinPool.java:1706)io.netty.util.internal.chmv8.ForkJoinPool.runWorker(ForkJoinPool.java:1661)io.netty.util.internal.chmv8.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:126)
#1:io.netty.buffer.AdvancedLeakAwareByteBuf.skipBytes(AdvancedLeakAwareByteBuf.java:465)io.netty.handler.codec.DelimiterBasedFrameDecoder.decode(DelimiterBasedFrameDecoder.java:272)io.netty.handler.codec.DelimiterBasedFrameDecoder.decode(DelimiterBasedFrameDecoder.java:216)io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:316)io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:230)io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:956)io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:127)io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:514)io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:471)io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:385)io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:351)io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)io.netty.util.internal.chmv8.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1412)io.netty.util.internal.chmv8.ForkJoinTask.doExec(ForkJoinTask.java:280)io.netty.util.internal.chmv8.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:877)io.netty.util.internal.chmv8.ForkJoinPool.scan(ForkJoinPool.java:1706)io.netty.util.internal.chmv8.ForkJoinPool.runWorker(ForkJoinPool.java:1661)io.netty.util.internal.chmv8.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:126)
Created at:io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:250)io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146)io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:107)io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:113)io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:514)io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:471)io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:385)io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:351)io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)io.netty.util.internal.chmv8.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1412)io.netty.util.internal.chmv8.ForkJoinTask.doExec(ForkJoinTask.java:280)io.netty.util.internal.chmv8.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:877)io.netty.util.internal.chmv8.ForkJoinPool.scan(ForkJoinPool.java:1706)io.netty.util.internal.chmv8.ForkJoinPool.runWorker(ForkJoinPool.java:1661)io.netty.util.internal.chmv8.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:126)

定位到我的代码中为：

ByteBuf buff=(ByteBuf) msg;
byte[] req=new byte[buff.readableBytes()];

于是可以确定是ByteBuff内存泄漏导致的问题，于是从这方面着手调查，发现netty5默认的分配bytebuff的方式是PooledByteBufAllocator,所以要手动回收，要不然会造成内存泄漏。
于是释放ByteBuff即可

ReferenceCountUtil.release(buff);

这里引入一个网友对于这行代码的说明：

ReferenceCountUtil.release()其实是ByteBuf.release()方法（从ReferenceCounted接口继承而来）的包装。netty4中的ByteBuf使用了引用计数（netty4实现了一个可选的ByteBuf池），每一个新分配的ByteBuf>>的引用计数值为1，每对这个ByteBuf对象增加一个引用，需要调用ByteBuf.retain()方法，而每减少一个引用，需要调用ByteBuf.release()方法。当这个ByteBuf对象的引用计数值为0时，表示此对象可回收。我这只是用ByteBuf说明，还有其他对象实现了ReferenceCounted接口，此时同理。

在检查问题的过程中，我还怀疑是不是我的Netty使用了UDP协议导致的数据丢失，于是这里附上Netty使用的是TCP还是UDP的判断方法:

关于TCP和UDP
socket可以基于TCP，也可以基于UDP。区别在于UDP的不保证数据包都正确收到，所以性能更好，但容错不高。TCP保证不错，所以性能没那么好。
UDP基本只适合做在线视频传输之类，我们的需求应该会是TCP。

那这2种方式在写法上有什么不同？网上搜到这样的说法：

在ChannelFactory 的选择上，UDP的通信选择 NioDatagramChannelFactory，TCP的通信我们选择的是NioServerSocketChannelFactory；
在Bootstrap的选择上，UDP选择的是ConnectionlessBootstrap，而TCP选择的是ServerBootstrap。

对于编解码器decoder和Encoder，以及ChannelPipelineFactory，UDP开发与TCP并没有什么区别，在此不做详细介绍。

对于ChannelHandler，是UDP与TCP区别的核心所在。大家都知道UDP是无连接的，也就是说你通过 MessageEvent 参数对象的 getChannel() 方法获取当前会话连接，但是其 isConnected() 永远都返回 false。
UDP 开发中在消息获取事件回调方法中，获取了当前会话连接 channel 对象后可直接通过 channel 的 write 方法发送数据给对端 channel.write(message, remoteAddress)，第一个参数仍然是要发送的消息对象，
第二个参数则是要发送的对端 SocketAddress 地址对象。
这里最需要注意的一点是SocketAddress，在TCP通信中我们可以通过channel.getRemoteAddress()获得，但在UDP通信中，我们必须从MessageEvent中通过调用getRemoteAddress()方法获得对端的SocketAddress 地址。

关于Netty的ByteBuff内存泄漏问题相关推荐

Netty防止内存泄漏措施
谨以此文献给李林锋即将新生的爱女. 1. 背景 1.1 直播平台内存泄漏问题某直播平台,一些网红的直播间在业务高峰期,会有 10W+ 的粉丝接入,如果瞬间发生大量客户端连接掉线.或者一些客户端网络 ...
haddler处理队列 netty_Netty堆外内存泄漏排查，这一篇全讲清楚了
上篇文章介绍了Netty内存模型原理,由于Netty在使用不当会导致堆外内存泄漏,网上关于这方面的资料比较少,所以写下这篇文章,专门介绍排查Netty堆外内存相关的知识点,诊断工具,以及排查思路提供参 ...
java nio 李林峰_Netty堆外内存泄漏排查，这一篇全讲清楚了
上篇文章介绍了Netty内存模型原理,由于Netty在使用不当会导致堆外内存泄漏,网上关于这方面的资料比较少,所以写下这篇文章,专门介绍排查Netty堆外内存相关的知识点,诊断工具,以及排查思路提供参 ...
Netty之有效规避内存泄漏
有过痛苦的经历,特别能写出深刻的文章 -- 凯尔文. 肖直接内存是IO框架的绝配,但直接内存的分配销毁不易,所以使用内存池能大幅提高性能,也告别了频繁的GC.但,要重新培养被Java的自动垃圾回收惯 ...
Netty技术细节源码分析-ByteBuf的内存泄漏原因与检测
本文的github地址:点此该文所涉及的netty源码版本为4.1.6. Netty中的ByteBuf为什么会发生内存泄漏在Netty中,ByetBuf并不是只采用可达性分析来对ByteBuf底层 ...
dotnetty java netty,『神坑』DotNetty 内存泄漏解决办法
背景近来在用 DotNetty 实现一个文件上传下载的同步服务. 其中:客户端下载服务端的文件,客户端多次请求,从服务端将文件分片下载下来,追加到本地磁盘. -- 非常简单的代码,都写了几十次了,驾 ...
java内存泄露direct_详解SpringCloudGateway内存泄漏问题
SpringCloudGateway内存泄漏问题项目完善差不多,在进入压力测试阶段期间,发现了gateway有内存泄漏问题,问题发现的起因是,当时启动一台gateway,一台对应的下游应用服务,在压 ...
利用classloader同一个项目中加载另一个同名的类_线程上下文类加载器ContextClassLoader内存泄漏隐患...
前提今天(2020-01-18)在编写Netty相关代码的时候,从Netty源码中的ThreadDeathWatcher和GlobalEventExecutor追溯到两个和线程上下文类加载器Cont ...
jstat 内存泄漏_一次Java内存泄漏的排查！要了自己的老命！
点击上方"Java之间",选择"置顶或者星标" 你关注的就是我关心的! 作者:枕边书来源:https://zhenbianshu.github.io 一.由来 ...

关于Netty的ByteBuff内存泄漏问题

关于Netty的ByteBuff内存泄漏问题相关推荐

最新文章

热门文章