feat(kafka): automatically upsize topic partition counts #15714

rob-1019 · 2025-12-17T21:38:28Z

Extends CreateKafkaTopicsStep to detect and automatically increase partition counts for existing topics that have fewer partitions than configured.

Previously, the upgrade step only created new topics and skipped existing ones. Now it checks existing topics and increases their partition count if they're under-provisioned. The step fails if partition count checks fail, ensuring operators are alerted to configuration issues. Note that Kafka does not support reducing partition counts, so topics with more partitions than configured are left unchanged with a warning.

Changes:

Add getCurrentPartitionCount() method to check existing partition counts
Track partitionsToIncrease map for topics that need upsizing
Track failedTopics list and throw exception if any checks fail
Add comprehensive test coverage for upsizing and failure scenarios
Update kafka-config.md to document automatic partition upsizing behavior
The PR conforms to DataHub's Contributing Guideline (particularly PR Title Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Extends CreateKafkaTopicsStep to detect and automatically increase partition counts for existing topics that have fewer partitions than configured. Previously, the upgrade step only created new topics and skipped existing ones. Now it checks existing topics and increases their partition count if they're under-provisioned. The step fails if partition count checks fail, ensuring operators are alerted to configuration issues. Note that Kafka does not support reducing partition counts, so topics with more partitions than configured are left unchanged with a warning. Changes: - Add getCurrentPartitionCount() method to check existing partition counts - Track partitionsToIncrease map for topics that need upsizing - Track failedTopics list and throw exception if any checks fail - Add comprehensive test coverage for upsizing and failure scenarios - Update kafka-config.md to document automatic partition upsizing behavior

… auto-upsize 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>

codecov · 2025-12-17T22:37:12Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

deepgarg760 · 2025-12-19T10:10:20Z

...b-upgrade/src/main/java/com/linkedin/datahub/upgrade/system/kafka/CreateKafkaTopicsStep.java

+                    desiredPartitions);
+                topicsToSkip.add(topicName);
+              } else {
+                log.info(


This logging is not required as this a default behavior before this change also

Cleaned this up a bit. PTAL.

...b-upgrade/src/main/java/com/linkedin/datahub/upgrade/system/kafka/CreateKafkaTopicsStep.java

deepgarg760 · 2025-12-19T10:15:05Z

...b-upgrade/src/main/java/com/linkedin/datahub/upgrade/system/kafka/CreateKafkaTopicsStep.java

+                      + "These topics may not have the correct partition configuration.",
+                  failedTopics.size(), failedTopics);
+          log.error(errorMessage);
+          throw new RuntimeException(errorMessage);


IMO, we should not throw exception here and log the warning is sufficient. It should not be blocker for topic creation

Feel pretty strongly that failure to configure these items during a setup job should be fatal if the user has this functionality enabled. Same for topic creation and most other things during setup jobs do - its better to fail loudly. Failures may get auto-triaged by restart attempts, quiet ignores can't be.

deepgarg760 · 2025-12-19T10:15:57Z

...b-upgrade/src/main/java/com/linkedin/datahub/upgrade/system/kafka/CreateKafkaTopicsStep.java

          if (existingTopics.contains(topicName)) {
-            log.info("Topic {} already exists - skipping creation", topicName);
-            topicsToSkip.add(topicName);
+            // For existing topics, check if partition count needs to be increased


IMO, this new block can have its own separate function for code readability

Take a look and let me know if that doesn't address core concern here. Made the log messages a bit more consistant to ease grepping for the various categories of outcome.

deepgarg760 · 2025-12-19T10:22:22Z

IMO, this should be controlled by feature flag like: DATAHUB_AUTO_INCREASE_PARTITIONS

deepgarg760 · 2025-12-19T10:27:04Z

...b-upgrade/src/main/java/com/linkedin/datahub/upgrade/system/kafka/CreateKafkaTopicsStep.java

-            topicsToSkip.add(topicName);
+            // For existing topics, check if partition count needs to be increased
+            try {
+              int currentPartitions = getCurrentPartitionCount(adminClient, topicName);


Instead of doing single API call for each existing, can this be batch. The goal is to have single API call for all existing topics

Automatically increases partition counts for existing Kafka topics when configured values exceed current counts during upgrades. Adds DATAHUB_AUTO_INCREASE_PARTITIONS flag for operator control. Key changes: - Add DATAHUB_AUTO_INCREASE_PARTITIONS environment variable (defaults to true) - Batch describe topics API call for efficiency (1 call vs N calls) - Extract fetchPartitionCountsForExistingTopics method for readability - Improve logging with consistent "Checking kafka topic" format - Change partition reduction log from WARN to ERROR level - Add comprehensive test coverage for auto-increase scenarios 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>

rob-1019 · 2025-12-19T17:01:08Z

IMO, this should be controlled by feature flag like: DATAHUB_AUTO_INCREASE_PARTITIONS

Excellent suggestion. Done. Overloading that was naughty.

Add new kafka.setup.autoIncreasePartitions property to the non-sensitive properties list in PropertiesCollectorConfigurationTest to fix test failure. This property controls automatic partition increases for Kafka topics and does not contain sensitive data. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>

codecov · 2025-12-19T18:58:30Z

Bundle Report

Changes will increase total bundle size by 5.35kB (0.02%) ⬆️. This is within the configured threshold ✅

Detailed changes

Bundle name	Size	Change
datahub-react-web-esm	31.28MB	5.35kB (0.02%) ⬆️

Affected Assets, Files, and Routes:

view changes for bundle: datahub-react-web-esm

Assets Changed:

Asset Name	Size Change	Total Size	Change (%)
`assets/index-*.js`	6.21kB	19.25MB	0.03%
`assets/index-*.css`	-857 bytes	608.59kB	-0.14%

Update error message in CreateKafkaTopicsStep to accurately reflect that failedTopics tracks all types of topic failures (creation, configuration, partition count checks), not just partition count verification. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>

github-actions bot added docs Issues and Improvements to docs community-contribution PR or Issue raised by member(s) of DataHub Community labels Dec 17, 2025

docs(updating): add note for PR datahub-project#15714 kafka partition…

4a0dac9

… auto-upsize 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Dec 17, 2025

vercel bot deployed to Preview December 17, 2025 21:53 View deployment

vercel bot deployed to Preview December 17, 2025 22:09 View deployment

deepgarg760 reviewed Dec 19, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Dec 19, 2025

deepgarg760 reviewed Dec 19, 2025

View reviewed changes

...b-upgrade/src/main/java/com/linkedin/datahub/upgrade/system/kafka/CreateKafkaTopicsStep.java Outdated Show resolved Hide resolved

deepgarg760 reviewed Dec 19, 2025

View reviewed changes

datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Dec 19, 2025

vercel bot deployed to Preview December 19, 2025 17:03 View deployment

vercel bot deployed to Preview December 19, 2025 18:23 View deployment

vercel bot deployed to Preview December 19, 2025 23:01 View deployment

feat(kafka): automatically upsize topic partition counts #15714

Are you sure you want to change the base?

feat(kafka): automatically upsize topic partition counts #15714

Conversation

rob-1019 commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 17, 2025

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deepgarg760 commented Dec 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rob-1019 commented Dec 19, 2025

Uh oh!

codecov bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bundle Report

Affected Assets, Files, and Routes:

Assets Changed:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rob-1019 commented Dec 17, 2025 •

edited

Loading

codecov bot commented Dec 19, 2025 •

edited

Loading