Netty leak detector logs "LEAK: ByteBuf.release() was not called before it's garbage-collected" in multiple unit test with BufferedChannel in the stack.

[dlg99]

BUG REPORT

1. Please describe the issue you observed:

related to issues #1610 and #1614

• What did you do?

ran unit tests with netty's paranoid leak detectot enabled.

• What did you expect to see?

no leak

• What did you see instead?

multiple log entries like "LEAK: ByteBuf.release() was not called before it's garbage-collected"

ERROR - [pool-77-thread-1:Slf4JLogger@171] - LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736)
org.apache.bookkeeper.bookie.BufferedChannel.flush(BufferedChannel.java:183)
org.apache.bookkeeper.bookie.BufferedChannel.flushAndForceWrite(BufferedChannel.java:157)
org.apache.bookkeeper.bookie.EntryLogManagerBase.flushLogChannel(EntryLogManagerBase.java:125)
org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog.flushCurrentLogs(EntryLogManagerForSingleEntryLog.java:170)
org.apache.bookkeeper.bookie.EntryLogManagerBase.flush(EntryLogManagerBase.java:119)
org.apache.bookkeeper.bookie.EntryLogger.flush(EntryLogger.java:601)
org.apache.bookkeeper.bookie.EntryLogger.shutdown(EntryLogger.java:1064)
org.apache.bookkeeper.bookie.InterleavedLedgerStorage.shutdown(InterleavedLedgerStorage.java:186)
org.apache.bookkeeper.bookie.SortedLedgerStorage.shutdown(SortedLedgerStorage.java:124)
org.apache.bookkeeper.bookie.Bookie.shutdown(Bookie.java:1106)
org.apache.bookkeeper.bookie.Bookie.shutdown(Bookie.java:1075)
org.apache.bookkeeper.proto.BookieServer.shutdown(BookieServer.java:201)
org.apache.bookkeeper.test.BookKeeperClusterTestCase.killBookie(BookKeeperClusterTestCase.java:406)

in multiple unit tests.

Maybe there are other stacks but ones I checked were all related to BufferedChannel.

[BramDeCneudt]

Is this issue still open? I also have the same issue but with an other stacktrace:

I'm using the old 0.5.0 version from distributed log, has this been patched in the later versions?

2018-11-14 10:28:00.425 ERROR [,,,] 1756 --- [ DL-io-1] io.netty.util.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: 0 Created at: io.netty.util.ResourceLeakDetector.track(ResourceLeakDetector.java:237) io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181) io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117) org.apache.distributedlog.EnvelopedEntryWriter.<init>(EnvelopedEntryWriter.java:83) org.apache.distributedlog.Entry.newEntry(Entry.java:52) org.apache.distributedlog.BKLogSegmentWriter.newRecordSetWriter(BKLogSegmentWriter.java:434) org.apache.distributedlog.BKLogSegmentWriter.abortTransmitPacketOnClose(BKLogSegmentWriter.java:554) org.apache.distributedlog.BKLogSegmentWriter.access$200(BKLogSegmentWriter.java:102) org.apache.distributedlog.BKLogSegmentWriter$3.onSuccess(BKLogSegmentWriter.java:524) org.apache.distributedlog.BKLogSegmentWriter$3.onSuccess(BKLogSegmentWriter.java:521) org.apache.distributedlog.common.concurrent.FutureEventListener.accept(FutureEventListener.java:42) org.apache.distributedlog.common.concurrent.FutureEventListener.accept(FutureEventListener.java:26) java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) org.apache.distributedlog.BKTransmitPacket.notifyTransmitComplete(BKTransmitPacket.java:54) org.apache.distributedlog.BKLogSegmentWriter.addComplete(BKLogSegmentWriter.java:1172) org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:311) org.apache.bookkeeper.client.LedgerHandle.sendAddSuccessCallbacks(LedgerHandle.java:1272) org.apache.bookkeeper.client.PendingAddOp.sendAddSuccessCallbacks(PendingAddOp.java:288) org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:283) org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion$1.writeComplete(PerChannelBookieClient.java:1890) org.apache.bookkeeper.proto.PerChannelBookieClient.handleAddResponse(PerChannelBookieClient.java:1580) org.apache.bookkeeper.proto.PerChannelBookieClient$15.safeRun(PerChannelBookieClient.java:1378) org.apache.bookkeeper.util.SafeRunnable.run(SafeRunnable.java:33) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)

[sijie]

@BramDeCneudt when did you observe this issue? at shutting down?

0.5.0 is pretty old version, I would suggest upgrading to 4.7.2 or 4.8.0

[BramDeCneudt]

@sijie thanks for the quick reply!
I always have this behaviour when starting up the application.

We are still using the version from the distributed log before it was ported to the bookkeeper repository but can't seem to find anywhere if the newer versions of bookkeeper are backwards compatible with the older versions and what changed on the distributed log repo during the multiple release of bookkeeper?

Is there somewhere some information that I can find about that?

[eolivelli]

@BramDeCneudt
there should be any "incompatibilty"

You'd better check the release notes of each release, but I bet you won't hit issues.
We are upgrading third party libraries, like Netty, so your classpath will change a bit

[BramDeCneudt]

@eolivelli Thanks for the reply, then I will first try to upgrade the version and see if the problems persists thanks!

[eolivelli]

@BramDeCneudt FYI we are going to release 4.8.1 maybe you can upgrade to that version. If you have time you can try the release candidate

[artus]

We still have the same issue on version 4.3.7

You can see the stacktrace here.