Skip to content

KAFKA-20792: Fix invalid partition -1 in FK join triggered by punctuator - #22980

Open
siddharthaDevineni wants to merge 1 commit into
apache:trunkfrom
siddharthaDevineni:KAFKA-20792-fk-join-invalid-partition
Open

KAFKA-20792: Fix invalid partition -1 in FK join triggered by punctuator#22980
siddharthaDevineni wants to merge 1 commit into
apache:trunkfrom
siddharthaDevineni:KAFKA-20792-fk-join-invalid-partition

Conversation

@siddharthaDevineni

Copy link
Copy Markdown
Contributor

StreamTask#punctuate sets a dummy ProcessorRecordContext (null topic, -1 partition and -1 offset), so records forwarded from a punctuator keep the punctuation timestamp. ProcessorContext#recordMetadata therefore returns a present-but-dummy Òtptional` rather than an empty one.

When a punctuator mutates the materialized store backing the left side of a foreign-key join and caching is enabled, the cache captures that dummy context and replays it when the change is flushed downstream. SubscriptionSendProcessorSupplier reads partition -1 into SubscriptionWrapper#primaryPartition, which was serialized to the subscription registration topic and later handed to the response sink partitioner, failing with ÌllegalArgumentException: Invalid partition: -1`.

This change uses null for primaryPartition when no valid source partition is available. null already means "unknown partition, use default partitioning": the deserializer sets it for pre-VERSION_1 wrappers, which is how FK joins routed responses before the field existed.

Since the V1 field is a fixed-width int with no way to express absence, null is encoded as -1 on the wire and mapped back on read. The byte layout is unchanged. So, it's like we encode null as a NULL_PRIMARY_PARTITION (-1), we decode this constant as null. So, this only assigns meaning to a previously impossible value (-1 was never a valid partition).

Testing

  • SubscriptionSendProcessorSupplierTest: two unit tests covering dummy record metadata (partition -1) and absent record metadata.
  • SubscriptionWrapperSerdeTest: round-trip of a V1 wrapper with a null primaryPartition.
  • KTableKTableForeignKeyJoinScenarioTest#shouldNotFailWhenPunctuatorDeletesFromLeftTableStore end-to-end testing using `TopologyTestDriver. It asserts both that no exception is thrown and that the leftJoin emits the expected tombstone. The test also fails without the production change, with the same stack trace reported in the Jira ticket.
  • shouldThrowExceptionOnNullPrimaryPartitionV1Test is removed. It asserted an auto-unboxing NPEfrom buf.putInt(data.primaryPartition()) rather than a deliberate validation: SubscriptionWrappers constructor guards other fields but not primaryPartition and null is already the meaningful V0 for that field.

@github-actions github-actions Bot added triage PRs from the community streams labels Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

streams triage PRs from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant