DirectMemoryCRC32Digest.update(ByteBuf) allocates lots of byte[]

[mtang01]

FEATURE REQUEST

1. Please describe the feature you are requesting.
   Profiling shows the code block in bold is hit every time an entry is read from bookie.

    @Override
    public void update(ByteBuf buf) {
        int index = buf.readerIndex();
        int length = buf.readableBytes();

        try {
            if (buf.hasMemoryAddress()) {
                // Calculate CRC directly from the direct memory pointer
                crcValue = (int) updateByteBuffer.invoke(null, crcValue, buf.memoryAddress(), index, length);
            } else if (buf.hasArray()) {
                // Use the internal method to update from array based
                crcValue = (int) updateBytes.invoke(null, crcValue, buf.array(), buf.arrayOffset() + index, length);
            } else {
                **// Fallback to data copy if buffer is not contiguous
                byte[] b = new byte[length];
                buf.getBytes(index, b, 0, length);
                crcValue = (int) updateBytes.invoke(null, crcValue, b, 0, b.length);**
            }
        } catch (IllegalAccessException | InvocationTargetException e) {
            throw new RuntimeException(e);
        }
    }

This is due to ByteBuf being marked as read only (as a result buf.hasArray() == false) from

ReadCompletion
public void handleV3Response(BookkeeperProtocol.Response response) {
            readEntryOutstanding.dec();
            ReadResponse readResponse = response.getReadResponse();
            StatusCode status = response.getStatus() == StatusCode.EOK
                ? readResponse.getStatus() : response.getStatus();
            ByteBuf buffer = Unpooled.EMPTY_BUFFER;
            if (readResponse.hasBody()) {
                **buffer = Unpooled.wrappedBuffer(readResponse.getBody().asReadOnlyByteBuffer());**
            }

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have).
   Are you currently using any workarounds to address this issue?

This generates lots of garbage. See attached image. Would be nice to get your thoughts on this issue, maybe using a buffer pool.

3. Provide any additional detail on your proposed use case for this feature.

[merlimat]

@mtang01 Can you get the class name of the ByteBuf instance?

[merlimat]

Also, you could try with Crc32C which shouldn't have that issue.

[mtang01]

@merlimat io.netty.buffer.ByteBuf used in DirectMemoryCRC32Digest.update for the entry read code path wraps java.nio.HeapByteBufferR at its core. I'll give crc32c a try. Thanks.

[mtang01]

crc32c has the same problem. It calls com.scurrilous.circe.impl.resume, which allocates new byte[] due to ByteBuffer being read only.

@Override
    public int resume(int current, ByteBuffer input) {
        final byte[] array;
        final int index;
        final int length = input.remaining();
        if (input.hasArray()) {
            array = input.array();
            index = input.arrayOffset() + input.position();
            input.position(input.limit());
        } else {
            array = new byte[length];
            index = 0;
            input.get(array);
        }
        return resumeUnchecked(current, array, index, length);
    }

[merlimat]

Ouch, yes, the problem is at the ByteBuffer level. For CRC32 it might be possible to just call CRC32.update(input.nioBuffer()) that should be handled by JDK and should do the right thing.

[mtang01]

CRC32.update(ByteBuffer) also allocates byte[]. The problem is a readonly ByteBuffer doesn't expose its byte[] data, which CRC needs.