Skip to content

Reduce allocations in PulsarEntryFormatter batch encoding#126

Open
Denovo1998 wants to merge 6 commits intodatastax:4.0_dsfrom
Denovo1998:denovo-kop
Open

Reduce allocations in PulsarEntryFormatter batch encoding#126
Denovo1998 wants to merge 6 commits intodatastax:4.0_dsfrom
Denovo1998:denovo-kop

Conversation

@Denovo1998
Copy link

Motivation

PulsarEntryFormatter currently builds an intermediate list of converted messages before serializing the final batch payload, which adds extra allocations and increases peak memory usage. In addition, the batch buffer starts with a small fixed size and is only released on the normal return path, which can lead to unnecessary buffer growth and potential ByteBuf leaks if an exception occurs. Streamlining the encode path reduces overhead and makes the transactional/control metadata handling easier to follow.

Modifications

  • Pre-size the batch buffer using AppendInfo.validBytes() (bounded by a maximum) to reduce reallocations and copies.
  • Serialize records into the batch payload in a single pass instead of materializing an intermediate message list.
  • Cache isTransactional / isControlBatch per record batch to reduce repeated checks and improve readability.
  • Guard batch size tracking/logging work behind log.isTraceEnabled() to avoid extra overhead in normal operation.
  • Always release the temporary batch buffer via try/finally to prevent leaks on error paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant