[Bug] KoP: brokerIndex Ordering Issues in Ledger Entries Due to Write Failure

[zhaizhibo]

Search before asking

• I searched in the issues and found nothing similar.

Read release policy

• I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

Version

OS: centos
Pulsar version: 3.3.3
KoP: 2.10.2.2

Minimal reproduce step

Request A calls interceptor.beforeAddEntry(), successfully obtains and assigns index index1, then proceeds with the write operation.

Request B arrives concurrently, calls interceptor.beforeAddEntry(), successfully obtains and assigns index index2, then proceeds with the write operation.

Request A fails, causing its write operation to roll back, and the index generator (indexGenerator) is rolled back.

Request B’s write operation succeeds, and index index2 is written into the ledger and retained.

Request C arrives, obtains index index3, and finds that index3 is smaller than index2.

What did you expect to see?

All ledger entry indexes should be strictly monotonically increasing, with no later index being smaller than a previous one.

Even if Request A fails and is rolled back, the index generator should not result in a non-monotonic sequence

What did you see instead?

After Request A fails and is rolled back, index2 is written and retained in the ledger as a valid index, causing Request C to obtain index3, which is smaller than index2. This results in a non-monotonic sequence of indexes.

This non-monotonic sequence causes issues with binary search operations, as they rely on the assumption of strictly increasing indexes and fail to correctly find entries by offset.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[zhaizhibo]

When performing a write operation to a single ledger in Pulsar, the KoP interceptor calls intercepter.beforeAddEntry() before the write completes, which increments the broker’s index for the corresponding ledger. For example, request A calls this method and gets index1, which is then written into the entry. If request B arrives concurrently, it also calls intercepter.beforeAddEntry() and gets index2, which is written into the entry.

The issue arises when request A fails and its indexGenerator is rolled back, but request B’s write operation succeeds, meaning the entry for request B still contains the incorrect index2. Furthermore, if a subsequent request C is processed and its index index3 turns out to be smaller than index2, the brokerIndex in entries within the same ledger will no longer be strictly increasing. This non-monotonic sequence can lead to unexpected behavior in KoP when performing binary searches for the entry corresponding to a given offset. This issue requires further investigation to ensure that ledger entries maintain a correct monotonically increasing index even in scenarios where write operations fail and rollbacks occur.

[lhotari]

@KernLC The issue you are describing is potentially related to #23940, #23983 and #23994 in Pulsar.

[lhotari]

@KernLC KoP is not part of Apache Pulsar project, so the bug itself is not in Pulsar. To fix the issue, it might require changes in Pulsar.