Add support for broker demotion #191

kyguy · 2026-01-13T03:15:15Z

Suggested solution

This feature would extend the KafkaRebalance resource to leverage Cruise Control’s /kafkacruisecontrol/demote_broker endpoint [1], allowing users to specify a list of brokers to be demoted. This would trigger the migration of partition leadership away from those brokers in preparation for decommissioning or maintenance.

Addresses issue discussed here [2]

[1] https://github.com/linkedin/cruise-control/wiki/REST-APIs#demote-a-list-of-brokers-from-the-kafka-cluster
[2] strimzi/strimzi-kafka-operator#11907

Signed-off-by: Kyle Liberti <[email protected]>

see-quick

Thanks for the proposal, I think it looks good, just a few nits/questions...

Out of curiosity, I didn't find it in the proposal but what happens when demotion is in progress but target broker fails in the middle of operation?

128-broker-demotion-support.md

Signed-off-by: Kyle Liberti <[email protected]>

kyguy · 2026-01-14T18:06:03Z

Thanks for the review @see-quick

Out of curiosity, I didn't find it in the proposal but what happens when demotion is in progress but target broker fails in the middle of operation?

I just added this note to the Validation and constraints section of the proposal:

"If a target broker fails while leadership is being transferred to it, all demotion operations involving that broker are aborted, and the source brokers remain the leaders for the affected partitions.
In this case, the overall demotion request continues on a best-effort with the proposed operations, transferring the leadership on brokers that are available."

Signed-off-by: Kyle Liberti <[email protected]>

fvaleri

Thanks @kyguy. I left some comments.

I think it is also important to document that broker demotion does NOT prevent new partition leaders to be scheduled on the selected broker.

128-broker-demotion-support.md

tinaselenge

Thanks for the proposal. Overall, it looks good to me. I left a few comments to clarify.

tinaselenge · 2026-01-19T11:02:17Z

128-broker-demotion-support.md

+| brokers                          | integer array    | List of ids of broker to be demoted in the cluster.                |
+| concurrentLeaderMovements        | integer | Upper bound of ongoing leadership swaps. Default is 1000.                   |
+| skipUrpDemotion                  | boolean | Whether to skip demoting leader replicas for under-replicated partitions.   |
+| excludeFollowerDemotion          | boolean | Whether to skip demoting follower replicas on the broker to be demoted.     |


what does demoting follower replicas mean?

It moves the ids of the demoted brokers to the end of the replica lists.

tinaselenge · 2026-01-19T11:08:23Z

128-broker-demotion-support.md

+* When an impossible demotion operation is requested for example demoting all brokers or transferring leadership from the only in-sync replica when the KafkaRebalance `spec.skipUrpDemotion` configuration is set to `false`, the demotion request will be rejected and the error will be reported in the `KafkaRebalance` status.
+
+* If a target broker fails while leadership is being transferred to it, all demotion operations involving that broker are aborted, and the source brokers remain the leaders for the affected partitions.
+In this case, the overall demotion request continues on a best-effort basis with with the remaining proposed operations, transferring the leadership on brokers that are available.


Does the request completes successfully when only some of the leadership were transferred?

128-broker-demotion-support.md

scholzj · 2026-01-19T19:19:08Z

128-broker-demotion-support.md

+# Broker demotion support via KafkaRebalance resource
+
+This proposal extends the `KafkaRebalance` custom resource to support broker demotion by integrating with Cruise Control's `/demote_broker` endpoint. 
+This would allow users to demote brokers, removing them from partition leadership eligibility, in preparation for maintenance, decommissioning, or other operational needs.


How does this work? I get it that it removes them from partition leadership. But how does it ensure they are not eligible to become leaders again?

Cruise Control has the broker ids marked as ineligible in memory and moves the broker ids to the end of the replica lists, then triggers a leadership election. Since leadership election prioritizes choosing the first broker id of the replica list (the preferred leader) as the leader, the demoted brokers are less likely to be elected as leaders.

But how does it ensure they are not eligible to become leaders again?

If a rebalance or demotion request is made the broker ids marked as ineligible in memory are excluded from having partitions moved to them or being listed first in the replica lists.

What happens when CC pod is restarted (upgrade, migration to different node, etc).? User will have to trigger "rebalance" proposal and approval to refresh in-memory data?

At this time yes, to refresh the in-memory demotion data after Cruise Control pod restart, a user would have to trigger another KafkaRebalance demotion request.

scholzj · 2026-01-19T19:21:17Z

128-broker-demotion-support.md

+**NOTE**: As part of this proposal, we will also add a `excludeRecentlyDemotedBrokers` field for the `full`, `add-brokers`, and `remove-brokers` KafkaRebalance modes to give users the ability to prevent to leader replicas to be moved to recently demoted brokers.
+When `excludeRecentlyDemotedBrokers` is set to `true`, a broker is considered demoted for the duration specified by the Cruise Control `demotion.history.retention.time.ms` server configuration.
+By default, this value is 1209600000 milleseconds (14 days) but is configurable in the `spec.cruiseControl.config` section of the `Kafka` custom resource.


How does this work? Where is the information about the demotion work?

Cruise Control stores the demotion information in memory (yes this is far from ideal). Cruise Control will make sure the ids of the demoted brokers are in the end of the replica lists and will exclude the brokers from being considered for partition movement when generating optimization proposals.

128-broker-demotion-support.md

scholzj · 2026-01-19T19:29:07Z

128-broker-demotion-support.md

+
+The implementation includes the following validation:
+
+* When `demote-brokers` mode is specified, the `brokers` field must be provided.


Will it be validated only in the broker code or also with CEL?

Was originally planning on validating the field in the operator code but this may change depending on the proposal surrounding the API changes discussed #191 (comment).

I'll link the separate proposal here when I have a draft ready and we can pick this up once that proposal and its implementation is sorted.

128-broker-demotion-support.md

Signed-off-by: Kyle Liberti <[email protected]>

kyguy · 2026-01-29T00:23:22Z

Thanks everyone for the reviews! I am putting this proposal on hold temporarily as we address some of the API concerns raised in the thread here: #191 (comment). We need devise a better path forward for the parameters that are used in conjunction with mode field of the KafkaRebalance resource, maybe by consolidating primitive fields into a generic config section to ensure the API can support future modes cleanly.

I am going to draft a separate proposal for those API changes and link it here. Once that proposal and its implementation is complete, we can continue this broker demotion proposal with a cleaner, more maintainable foundation.

Signed-off-by: Kyle Liberti <[email protected]>

ppatierno · 2026-02-01T17:19:55Z

128-broker-demotion-support.md

+
+Broker demotion is independent of other rebalance modes but can be used before or after them manually:
+
+* **add-brokers**: After new brokers are added to the cluster, broker demotion could be used to explicitly transfer partition leadership away from existing brokers to accelerate leadership adoption on newly added brokers. 


I am not sure about this use case, isn't the auto-rebalancing (we support) already doing something like this? It's moving partitions but not demoting brokers.

From what I understand, auto-rebalancing and the add-brokers modes will move partition replicas onto the added brokers but not necessarily transfer partition leadership.

The use case I have in mind is one where a user wants to gradually transfer traffic to a new set of brokers and eventually remove the old brokers but is not ready to decommission them yet. The user can first rebalance the existing load onto the new brokers using add-brokers mode and then demote the old brokers using demote-brokers mode to shift the bulk of the traffic (the leadership) to the new brokers while still relying on the old brokers to host follower replicas.

Does that make sense?

Yes It does, thanks.

kyguy mentioned this pull request Jan 13, 2026

[Enhancement]: Add support for broker demotion strimzi/strimzi-kafka-operator#11907

Open

kyguy force-pushed the kr-demote-brokers branch 2 times, most recently from bbb1dc1 to 7fc8acd Compare January 13, 2026 22:08

Add support for broker demotion

abb52f1

Signed-off-by: Kyle Liberti <[email protected]>

kyguy force-pushed the kr-demote-brokers branch from 7fc8acd to abb52f1 Compare January 13, 2026 22:11

kyguy marked this pull request as ready for review January 13, 2026 22:12

see-quick reviewed Jan 14, 2026

View reviewed changes

128-broker-demotion-support.md Outdated Show resolved Hide resolved

128-broker-demotion-support.md Outdated Show resolved Hide resolved

128-broker-demotion-support.md Outdated Show resolved Hide resolved

128-broker-demotion-support.md Outdated Show resolved Hide resolved

Addressing feedback - mo

07cf465

Signed-off-by: Kyle Liberti <[email protected]>

kyguy force-pushed the kr-demote-brokers branch from d308e3a to 07cf465 Compare January 14, 2026 18:06

More revisions

f2f77b9

Signed-off-by: Kyle Liberti <[email protected]>

kyguy force-pushed the kr-demote-brokers branch from 7d66426 to f2f77b9 Compare January 14, 2026 21:33

kyguy requested review from Frawless, ShubhamRwt, fvaleri, im-konge, katheris, mimaison, ppatierno, scholzj, tinaselenge and tomncooper January 14, 2026 21:37

fvaleri reviewed Jan 15, 2026

View reviewed changes

tinaselenge reviewed Jan 19, 2026

View reviewed changes

scholzj reviewed Jan 19, 2026

View reviewed changes

ppatierno reviewed Jan 23, 2026

View reviewed changes

128-broker-demotion-support.md Outdated Show resolved Hide resolved

kyguy added 2 commits January 27, 2026 19:50

Addressing some feedback - fv

2c01570

Signed-off-by: Kyle Liberti <[email protected]>

Addressing feedback - fv, ts, js, pp

6a84e7f

Signed-off-by: Kyle Liberti <[email protected]>

Addressing feedback - fv

a302d54

Signed-off-by: Kyle Liberti <[email protected]>

ppatierno reviewed Feb 1, 2026

View reviewed changes


		The implementation includes the following validation:

		* When `demote-brokers` mode is specified, the `brokers` field must be provided.


		Broker demotion is independent of other rebalance modes but can be used before or after them manually:

		* add-brokers: After new brokers are added to the cluster, broker demotion could be used to explicitly transfer partition leadership away from existing brokers to accelerate leadership adoption on newly added brokers.

Add support for broker demotion #191

Are you sure you want to change the base?

Add support for broker demotion #191

Uh oh!

Conversation

kyguy commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related problem

Suggested solution

Uh oh!

see-quick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kyguy commented Jan 14, 2026

Uh oh!

fvaleri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tinaselenge left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kyguy Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kyguy Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kyguy Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kyguy commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kyguy commented Jan 13, 2026 •

edited

Loading

kyguy Jan 28, 2026 •

edited

Loading

kyguy Jan 28, 2026 •

edited

Loading

kyguy Jan 28, 2026 •

edited

Loading

kyguy commented Jan 29, 2026 •

edited

Loading