KAFKA-20430 Add configuration to control internal topic creation in Kafka Connect by anton-liauchuk · Pull Request #20384 · apache/kafka

anton-liauchuk · 2025-08-20T12:32:48Z

Implementation of
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209:+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect

Reviewers: Ken Huang s7133700@gmail.com, Chia-Ping Tsai
chia7712@gmail.com

anton-liauchuk · 2025-08-21T17:52:44Z

@C0urante, @mimaison, @chia7712 could you take a look at this POC?

mimaison · 2025-08-25T10:48:44Z

Hi @anton-liauchuk,

Your PoC introduces a new configuration so it requires a KIP. See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals for details about the process.

anton-liauchuk · 2025-08-25T12:55:27Z

Hi @anton-liauchuk,

Your PoC introduces a new configuration so it requires a KIP. See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals for details about the process.

hi @mimaison
thank you! I will be working on the KIP

anton-liauchuk · 2025-08-26T13:43:59Z

Hi @anton-liauchuk,

Your PoC introduces a new configuration so it requires a KIP. See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals for details about the process.

@mimaison KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209%3A+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect is created, please review

github-actions · 2025-08-28T03:21:58Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

github-actions · 2025-09-04T03:15:23Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

github-actions · 2025-09-11T03:19:30Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

github-actions · 2025-09-14T03:20:30Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

github-actions · 2025-12-13T03:47:17Z

This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact).

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.

github-actions · 2025-12-15T03:39:03Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

github-actions · 2026-03-15T04:19:04Z

This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact).

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.

github-actions · 2026-04-15T04:06:39Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

hgeraldino · 2026-05-15T18:34:13Z

looks good

github-actions · 2026-05-17T04:30:50Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

m1a2st

Thanks for the patch, some comments left

chia7712

@anton-liauchuk thanks for this patch. Please update upgrade.md as well

anton-liauchuk · 2026-07-28T09:55:27Z

@anton-liauchuk thanks for this patch. Please update upgrade.md as well

@chia7712
Added a note to the Notable changes in 4.4.0 section, thanks!

…tion # Conflicts: # docs/getting-started/upgrade.md

chia7712 · 2026-07-28T16:04:32Z

+                log.debug("Skipping creation of Connect internal topic for {} because automatic topic creation is disabled", getTopicPurpose());
+                Map<String, TopicDescription> existing = admin.describeTopics(topic);
+                if (existing.isEmpty()) {
+                    String msg = String.format("Topic '%s' specified via the '%s' property is missing." +


Is this error message correct if it's a connector-specific offset topic? Also, what getTopicConfig() returns wouldn't be the correct config key in that case, right?

m1a2st · 2026-07-28T12:33:19Z

  * Brokers can now record a human-readable description of each streams group's processing topology via a pluggable backend, retrievable through `Admin#describeStreamsGroups` and `kafka-streams-groups.sh --describe --topology`. The feature is disabled unless the new broker configuration `group.streams.topology.description.plugin.class` is set to a `StreamsGroupTopologyDescriptionPlugin` implementation; on the client side, the new Kafka Streams configuration `topology.description.push.enabled` (default `true`) controls whether the client pushes topology descriptions when requested. This adds a new RPC, `StreamsGroupTopologyDescriptionUpdate`, bumps `StreamsGroupDescribe` and `StreamsGroupHeartbeat` to version 1, and introduces the error codes `GROUP_DELETION_FAILED` (134) and `STREAMS_TOPOLOGY_DESCRIPTION_UPDATE_FAILED` (135). `DeleteGroups` is bumped to version 3, adding a per-group `ErrorMessage` field so brokers can report why a group deletion failed (for example, when the plugin fails to delete its stored topology description, the group is not deleted and `GROUP_DELETION_FAILED` is returned; retrying the deletion is safe). For further details, please refer to [KIP-1331](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1331%3A+Streams+Group+Topology+Description+Plugin) and the [Topology Description Plugin](/{version}/streams/developer-guide/topology-description-plugin/) documentation.
  * The `kafka-producer-perf-test.sh` tool now supports `--record-key-range`, `--key-distribution`, and `--random-seed` options to control the distribution of record keys. Use `--key-distribution range` for sequential key assignment (round-robin over the key range) or `--key-distribution random` for random key selection. The `--random-seed` option allows reproducible benchmark runs when using random key distribution. For further details, please refer to [KIP-1299](https://cwiki.apache.org/confluence/x/XpQ8G).
  * Share groups now support dead-letter queue functionality as outlined in [KIP-1191](https://cwiki.apache.org/confluence/x/fApJFg). Any records which are released (beyond max delivery count) or rejected by the share consumer become eligible for DLQ. Share group DLQ gets enabled when the Kafka feature `share.version` is upgraded to 2. The user can configure a DLQ topic on a share group by setting the dynamic config `errors.deadletterqueue.topic.name` (default `""`) to the name of the DLQ topic. The cluster can be configured to auto create the DLQ topics by setting the dynamic cluster config `errors.deadletterqueue.auto.create.topics.enable` to `true` (default `false`). If auto create is not enabled, the user must create the DLQ topic like a standard Kafka topic and set the dynamic config `errors.deadletterqueue.group.enable` to `true` on the DLQ topic. The DLQ topic name must be prefixed by the value set in the dynamic cluster config `errors.deadletterqueue.topic.name.prefix` (default `dlq.`). The records sent to the DLQ topic by default only contain source record metadata like group, topic name, partition id, offset and delivery count. If original record data is also required, the user must set the dynamic config `errors.deadletterqueue.copy.record.enable` to `true` on the share group.
+  * Kafka Connect distributed workers now support the `internal.topics.automatic.creation.enable` configuration (default: `true`). When set to `false`, Connect will not automatically create internal topics (offset, config, status, and connector-specific offset topics) and will instead fail at startup if any of these topics are missing. A new `connect-internal-topics.sh` tool is also available for manually creating these topics. For further details, please refer to [KIP-1209](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1209:+Add+configuration+to+control+internal+topic+creation+in+Kafka+Connect).


We should use immutable KIP link https://cwiki.apache.org/confluence/x/GAq2Fg

chia7712

LGTM and we will create the patch to fix the remaining tiny issues

anton-liauchuk · 2026-07-29T10:09:57Z

LGTM and we will create the patch to fix the remaining tiny issues

thanks a lot!

POC: conditional creation of internal topics

a60d5c0

github-actions Bot added triage PRs from the community connect small Small PRs labels Aug 20, 2025

github-actions Bot added the needs-attention label Aug 28, 2025

anton-liauchuk added 2 commits September 2, 2025 11:40

KIP-1209 verify topic config when auto creation is disabled

98670aa

KIP-1209 rename config

1fceb32

github-actions Bot removed the needs-attention label Sep 3, 2025

github-actions Bot added the needs-attention label Sep 4, 2025

KIP-1209 add integration test

b539b5b

github-actions Bot removed the needs-attention label Sep 10, 2025

github-actions Bot added the needs-attention label Sep 11, 2025

KIP-1209 rename config, improve error message

bad21aa

github-actions Bot removed the needs-attention label Sep 13, 2025

github-actions Bot added the needs-attention label Sep 14, 2025

github-actions Bot added stale Stale PRs and removed needs-attention stale Stale PRs labels Dec 13, 2025

github-actions Bot added the needs-attention label Dec 15, 2025

KIP-1209 change log level

daa516f

github-actions Bot removed the needs-attention label Apr 14, 2026

github-actions Bot added the needs-attention label Apr 15, 2026

github-actions Bot removed the needs-attention label May 16, 2026

github-actions Bot added the needs-attention label May 17, 2026

hgeraldino reviewed Jun 5, 2026

View reviewed changes

github-actions Bot removed needs-attention triage PRs from the community labels Jun 6, 2026

anton-liauchuk added 2 commits July 23, 2026 10:24

KIP-1209 default constants, test assertions

3e147ec

Merge branch 'apache:trunk' into internal_topics_creation

62d5e3b

anton-liauchuk requested a review from hgeraldino July 23, 2026 07:36

chia7712 added the ci-approved label Jul 23, 2026

m1a2st reviewed Jul 24, 2026

View reviewed changes

anton-liauchuk added 2 commits July 27, 2026 11:26

KIP-1209 documentation, rename constants

368aebb

KIP-1209 private constant

2e6af81

anton-liauchuk requested a review from m1a2st July 27, 2026 08:34

chia7712 reviewed Jul 27, 2026

View reviewed changes

Comment thread ...ct/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedConfig.java Outdated

Comment thread ...ct/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedConfig.java Outdated

KIP-1209 upgrade notes, config importance

72924c6

Merge remote-tracking branch 'origin/trunk' into internal_topics_crea…

5270a7a

…tion # Conflicts: # docs/getting-started/upgrade.md

anton-liauchuk requested a review from chia7712 July 28, 2026 09:58

chia7712 reviewed Jul 28, 2026

View reviewed changes

m1a2st reviewed Jul 28, 2026

View reviewed changes

chia7712 approved these changes Jul 29, 2026

View reviewed changes

chia7712 merged commit b992620 into apache:trunk Jul 29, 2026
24 checks passed

kaijaytu mentioned this pull request Jul 29, 2026

KAFKA-20430 Fix getTopicConfig() for connector-specific offset stores #22984

Open

Uh oh!

Conversation

anton-liauchuk commented Aug 20, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anton-liauchuk commented Aug 21, 2025

Uh oh!

mimaison commented Aug 25, 2025

Uh oh!

anton-liauchuk commented Aug 25, 2025

Uh oh!

anton-liauchuk commented Aug 26, 2025

Uh oh!

github-actions Bot commented Aug 28, 2025

Uh oh!

github-actions Bot commented Sep 4, 2025

Uh oh!

github-actions Bot commented Sep 11, 2025

Uh oh!

github-actions Bot commented Sep 14, 2025

Uh oh!

github-actions Bot commented Dec 13, 2025

Uh oh!

github-actions Bot commented Dec 15, 2025

Uh oh!

github-actions Bot commented Mar 15, 2026

Uh oh!

github-actions Bot commented Apr 15, 2026

Uh oh!

hgeraldino commented May 15, 2026

Uh oh!

github-actions Bot commented May 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

m1a2st left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

anton-liauchuk commented Jul 28, 2026

Uh oh!

Uh oh!

chia7712 Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

m1a2st Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

anton-liauchuk commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

anton-liauchuk commented Aug 20, 2025 •

edited by github-actions Bot

Loading