IllegalReferenceCountException while reading the data from ByteBuf

[joyfulnikhil]

Bug Description

We have a Spring Cloud Gateway Application which uses Netty as a server to process request/response. We see a spike of IllegalReferenceCountException happening on a particular TCP connection, not on any other Connections. Traffic on other connection work absolutely fine.

Below is the complete stack trace:

io.netty.util.IllegalReferenceCountException: refCnt: 0 errorCause: refCnt: 0 extended errorStackTrace: io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1454),io.netty.buffer.AbstractByteBuf.checkNewCapacity(AbstractByteBuf.java:1432),io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:105),io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:305),io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:280),io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1103),io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:105),io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:288),io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444),io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420), errorStackTrace: io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:294),io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444),io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420),io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412),io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93),ocpm.cne.gateway.util.ConnectionCountHandler.channelRead(ConnectionCountHandler.java:60),io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444),io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420),io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412),io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)

We would some guidance to understand what could be the reason that it only happens on one connection , while requests on other connections seems to work fine.

Netty version

4.1.109.Final

JVM version (e.g. java -version)

Java 17

[spencergibb]

What version of spring boot and spring cloud? What does your configuration look like? Can you describe how you are seeing what connection is having errors?

[csterwa]

@joyfulnikhil As asked in @spencergibb previous comment, could you provide more details on versions of libraries used and the configuration? Thank you.

[sukawanth]

@spencergibb @csterwa,

The issue is reproducible across different Spring Boot versions (2.7.x, 3.0.x, and 3.1.x). We are facing this issue with HttpProtocol.H2C and HttpProtocol.H2 ,but only one will be enabled at a time.

In exceptionCaught method of the last inbound handler in the pipeline, we are printing the remoteaddress when IllegalReferenceCountException occurs. All occurrences of the exception happen on the same thread and same connection.

[csterwa]

@spencergibb @csterwa,

The issue is reproducible across different Spring Boot versions (2.7.x, 3.0.x, and 3.1.x). We are facing this issue with HttpProtocol.H2C and HttpProtocol.H2 ,but only one will be enabled at a time.

In exceptionCaught method of the last inbound handler in the pipeline, we are printing the remoteaddress when IllegalReferenceCountException occurs. All occurrences of the exception happen on the same thread and same connection.

@sukawanth Do you have an example configuration and/or Boot application that reproduces this issue?

[lowsury]

I encountered the same problem, using version: springcloud-gateway-core: 2.2.5-RELEASE jdk: 1.8.0_391

Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1 at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74) at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138) at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100) at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88) at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:91) at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702) at reactor.netty.channel.MonoSendMany$SendManyInner.run(MonoSendMany.java:317) at reactor.netty.channel.MonoSendMany$SendManyInner.trySchedule(MonoSendMany.java:419) at reactor.netty.channel.MonoSendMany$SendManyInner.onNext(MonoSendMany.java:219) at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121) at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:96) at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2357) at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.request(FluxContextStart.java:125) at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162) at reactor.netty.channel.MonoSendMany$SendManyInner.onSubscribe(MonoSendMany.java:250) at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90) at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onSubscribe(FluxContextStart.java:90) at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) at reactor.core.publisher.Flux.subscribe(Flux.java:8357) at reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:102) at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:153) at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) at reactor.core.publisher.Mono.subscribe(Mono.java:4252) at reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:336) at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:65) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:446) at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:537) at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onStateChange(PooledConnectionProvider.java:563) at reactor.netty.resources.PooledConnectionProvider$PooledConnection.onStateChange(PooledConnectionProvider.java:450) at reactor.netty.channel.ChannelOperationsHandler.channelActive(ChannelOperationsHandler.java:65) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:209) at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelActive(CombinedChannelDuplexHandler.java:412) at io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:69) at io.netty.channel.CombinedChannelDuplexHandler.channelActive(CombinedChannelDuplexHandler.java:211) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:209) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1398) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) at io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:895) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:621) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:654) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:530) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:470) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

[csterwa]

@lowsury Could you provide sample of SCG server and also a client that reproduces this? I have not seen this error in any SCG instance thus far. Providing both samples (server and client) along with any other configurations that would be useful for reproducing this. Thank you.