Skip to content

KAFKA-20430 Fix getTopicConfig() for connector-specific offset stores - #22984

Open
kaijaytu wants to merge 1 commit into
apache:trunkfrom
kaijaytu:fix-connector-offset-getTopicConfig
Open

KAFKA-20430 Fix getTopicConfig() for connector-specific offset stores#22984
kaijaytu wants to merge 1 commit into
apache:trunkfrom
kaijaytu:fix-connector-offset-getTopicConfig

Conversation

@kaijaytu

@kaijaytu kaijaytu commented Jul 29, 2026

Copy link
Copy Markdown

Follow-up fix for #20384.

When internal.topics.automatic.creation.enable is set to false and a
connector-specific offset topic is missing, the error message
incorrectly references the worker-level config offset.storage.topic
instead of the connector-level config offsets.storage.topic. This is
because readWriteStore() and readOnlyStore() create anonymous
subclasses of KafkaOffsetBackingStore that do not override
getTopicConfig().

This patch overrides getTopicConfig() in both factory methods to
return SourceConnectorConfig.OFFSETS_TOPIC_CONFIG
(offsets.storage.topic), so the error message correctly directs users
to the relevant connector configuration.

Changes

  • KafkaOffsetBackingStore.java: Override getTopicConfig() in
    readWriteStore() and readOnlyStore() anonymous classes to return the
    connector-level config key
  • Added KafkaOffsetBackingStoreGetTopicConfigTest.java to verify the
    fix

Testing

Unit tests verify that:

  1. Worker-level KafkaOffsetBackingStore returns offset.storage.topic
    (unchanged behavior)
  2. readWriteStore() now returns offsets.storage.topic (was
    incorrectly returning offset.storage.topic)
  3. readOnlyStore() now returns offsets.storage.topic (was
    incorrectly returning offset.storage.topic)

Existing tests (KafkaOffsetBackingStoreTest,
ConnectorOffsetBackingStoreTest) pass with no regression.

@kaijaytu

Copy link
Copy Markdown
Author

@chia7712 This addresses the review comment from #20384 regarding the incorrect config key in error messages for connector-specific offset topics.

@github-actions github-actions Bot added connect triage PRs from the community labels Jul 29, 2026

@m1a2st m1a2st left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also address this comment #20384 (comment)

@kaijaytu
kaijaytu force-pushed the fix-connector-offset-getTopicConfig branch from 4cc3be1 to 68da24b Compare July 29, 2026 08:28
@kaijaytu
kaijaytu force-pushed the fix-connector-offset-getTopicConfig branch from 68da24b to c3a7433 Compare July 29, 2026 08:31
@kaijaytu

Copy link
Copy Markdown
Author

Addressed. This PR now also fixes the immutable KIP link in upgrade.md as suggested by @m1a2st in #20384.

@kaijaytu

Copy link
Copy Markdown
Author

Note: the error message in KafkaTopicBasedBackingStore also says "before starting the worker", which isn't accurate for connector-specific offset topics (they are checked at connector startup). Should I also fix the error message wording, or is that better addressed in a separate patch?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-approved connect triage PRs from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants