Add support for partition based scaling on the kafka scaler #6558

patelvp · 2025-02-20T06:26:44Z

Scaling kafka consumers should be done in factors of partition count on the topic. This is to ensure that the partitions are evenly spread across all consumers. If the paritions are not evenly spread we run the risk of some partitions being consumed faster than other. This PR adds a new property on the kafka scaler ensureEvenDistributionOfPartitions. When this property is set to true the scaler ensure that the number of pods are always evenly spread across the number of topics on the partition.

Checklist

When introducing a new scaler, I agree with the scaling governance policy
I have verified that my change is according to the deprecations & breaking changes policy
Tests have been added
Changelog has been updated and is aligned with our changelog requirements
A PR is opened to update the documentation on (repo) (if applicable)
Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #2581

QA:
Tested this locally by pushing an image on a local kind cluster. Kafka consumer and producers are outside the cluster to get granual and quick control over production and consumption rate.
Plotted the pod count and kafka partition lag onto grafana.

JorTurFer · 2025-02-20T21:26:40Z

@dttung2905 @zroubalik , you are the Kafka experts xD
Does this make sense?

dttung2905

Thank you very much for this PR. Personally, I like the direction this PR is heading to. Just 1 small comment for my understanding

pkg/scalers/kafka_scaler.go

zroubalik · 2025-02-25T09:08:58Z

/run-e2e kafka
Update: You can check the progress here

patelvp · 2025-03-03T16:13:18Z

@zroubalik Any thoughts on the PR?

patelvp · 2025-03-06T21:27:28Z

Force pushed a rebase

dttung2905 · 2025-03-08T10:17:37Z

/run-e2e kafka
Update: You can check the progress here

utkuaydn · 2025-04-15T11:27:29Z

Would it be possible to add this functionality to the experimental kafka trigger as well?

patelvp · 2025-04-16T00:48:29Z

Would it be possible to add this functionality to the experimental kafka trigger as well?

Yeah I could take a look at it.

rickbrouwer · 2025-06-15T18:40:03Z

/run-e2e kafka
Update: You can check the progress here

patelvp · 2025-06-20T00:23:16Z

@rickbrouwer I don't see the logs for the failed test. Can you share how I can see them?

rickbrouwer · 2025-06-20T05:26:33Z

@rickbrouwer I don't see the logs for the failed test. Can you share how I can see them?

Of course!
You can find the log here:

https://github.com/kedacore/keda/actions/runs/15666250049/job/44131008541#step:9:1886

patelvp · 2025-08-20T20:42:35Z

@rickbrouwer I don't see the logs for the failed test. Can you share how I can see them?

Of course! You can find the log here:

https://github.com/kedacore/keda/actions/runs/15666250049/job/44131008541#step:9:1886

@rickbrouwer The tests now pass with a rebase. Could it be possible it was a flake?
I can push a new rebase again

zroubalik

@patelvp could you please fix conflicts and fix the static checks? I will trigger the e2e tests then.

Thanks!

rickbrouwer · 2025-08-26T18:23:10Z

@patelvp there is still an issue in the static checks

rickbrouwer · 2025-08-27T04:29:22Z

/run-e2e kafka
Update: You can check the progress here

patelvp · 2025-08-27T18:59:47Z

Looking at the test failures

patelvp · 2025-08-28T05:35:52Z

Okay, I tried the test locally. It took me a while to figure out that our corporate VPN was blocking some image pulls for the test to run correctly. The fix was an indentation issue in the test template!

rickbrouwer · 2025-08-28T06:09:31Z

/run-e2e kafka
Update: You can check the progress here

patelvp · 2025-08-28T22:15:12Z

@rickbrouwer @zroubalik Should be good to review. Hoping I can make it in the next release so I don't have to keep maintaining this PR

Copilot

Pull Request Overview

This PR adds support for partition-based scaling to the Kafka scaler through a new ensureEvenDistributionOfPartitions property. When enabled, this feature ensures that the number of Kafka consumer pods aligns with factors of the topic's partition count to guarantee even partition distribution across consumers.

Key changes include:

Addition of the ensureEvenDistributionOfPartitions configuration option to both Kafka and Apache Kafka scalers
Implementation of mathematical logic to calculate appropriate scaling factors based on partition counts
Comprehensive test coverage for the new functionality

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
pkg/scalers/kafka_scaler.go	Implements core logic for partition-based scaling and factor calculation
pkg/scalers/apache_kafka_scaler.go	Adds the new configuration option to Apache Kafka scaler
pkg/scalers/kafka_scaler_test.go	Adds comprehensive unit tests for metadata parsing and factor calculations
pkg/scalers/apache_kafka_scaler_test.go	Updates test data structures to include new configuration option
tests/scalers/kafka/kafka_test.go	Adds integration tests for the new scaling behavior
tests/scalers/apache_kafka/apache_kafka_test.go	Adds integration tests for Apache Kafka scaler
CHANGELOG.md	Documents the new feature addition

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

pkg/scalers/kafka_scaler_test.go

tests/scalers/kafka/kafka_test.go

tests/scalers/apache_kafka/apache_kafka_test.go

tests/scalers/kafka/kafka_test.go

tests/scalers/apache_kafka/apache_kafka_test.go

tests/scalers/kafka/kafka_test.go

tests/scalers/apache_kafka/apache_kafka_test.go

zroubalik · 2025-09-18T09:22:34Z

@rickbrouwer @zroubalik Should be good to review. Hoping I can make it in the next release so I don't have to keep maintaining this PR

@patelvp could you please check the review comments?

dttung2905 · 2025-09-18T11:12:27Z

yes. imo, once all the comments are addresssed, it should be good to merge

zroubalik · 2025-09-30T14:11:51Z

@patelvp could you please resolve the comments so we can include this PR into the next release

patelvp · 2025-09-30T15:27:10Z

@patelvp could you please resolve the comments so we can include this PR into the next release

I will get to this today

Scaling kafka consumers should be done in factors of partition count on the topic. This is to ensure that the partitions are evenly spread across all consumers. If the paritions are not evenly spread we run the risk of some partitions being consumed faster than other. This PR adds a new property on the kafka scaler `ensureEvenDistributionOfPartitions`. When this property is set to true the scaler ensure that the number of pods are always evenly spread across the number of topics on the partition. Signed-off-by: Vishal Patel <[email protected]>

patelvp · 2025-10-01T19:59:30Z

@zroubalik Should be good now!

rickbrouwer · 2025-10-02T05:20:59Z

/run-e2e kafka
Update: You can check the progress here

…#6558) Scaling kafka consumers should be done in factors of partition count on the topic. This is to ensure that the partitions are evenly spread across all consumers. If the paritions are not evenly spread we run the risk of some partitions being consumed faster than other. This PR adds a new property on the kafka scaler `ensureEvenDistributionOfPartitions`. When this property is set to true the scaler ensure that the number of pods are always evenly spread across the number of topics on the partition. Signed-off-by: Vishal Patel <[email protected]> Signed-off-by: Dmitriy Altuhov <[email protected]>

…#6558) Scaling kafka consumers should be done in factors of partition count on the topic. This is to ensure that the partitions are evenly spread across all consumers. If the paritions are not evenly spread we run the risk of some partitions being consumed faster than other. This PR adds a new property on the kafka scaler `ensureEvenDistributionOfPartitions`. When this property is set to true the scaler ensure that the number of pods are always evenly spread across the number of topics on the partition. Signed-off-by: Vishal Patel <[email protected]>

patelvp requested a review from a team as a code owner February 20, 2025 06:26

patelvp mentioned this pull request Feb 20, 2025

Kafka Partitions Per Pod Scaling #2581

Closed

patelvp force-pushed the add-partition-based-scaling-kafka-scaler branch from a548ac9 to c46e703 Compare February 20, 2025 17:37

dttung2905 reviewed Feb 20, 2025

View reviewed changes

pkg/scalers/kafka_scaler.go Show resolved Hide resolved

patelvp force-pushed the add-partition-based-scaling-kafka-scaler branch from c46e703 to a7ad359 Compare March 6, 2025 21:27

zroubalik approved these changes Mar 20, 2025

View reviewed changes

zroubalik mentioned this pull request Mar 26, 2025

Release: 2.17.0 #6502

Closed

40 tasks

patelvp force-pushed the add-partition-based-scaling-kafka-scaler branch 2 times, most recently from b18b0d1 to 56e8e43 Compare June 13, 2025 18:01

patelvp requested a review from zroubalik June 17, 2025 19:19

patelvp force-pushed the add-partition-based-scaling-kafka-scaler branch from 56e8e43 to 41591d7 Compare August 11, 2025 01:01

zroubalik mentioned this pull request Aug 19, 2025

Release: 2.18.0 #6853

Closed

22 tasks

zroubalik reviewed Aug 26, 2025

View reviewed changes

patelvp force-pushed the add-partition-based-scaling-kafka-scaler branch from 41591d7 to 2eef3fc Compare August 26, 2025 17:49

patelvp requested a review from a team as a code owner August 26, 2025 17:49

keda-automation requested a review from a team August 26, 2025 17:50

patelvp force-pushed the add-partition-based-scaling-kafka-scaler branch from 2eef3fc to ad2c2d9 Compare August 26, 2025 23:36

patelvp force-pushed the add-partition-based-scaling-kafka-scaler branch from ad2c2d9 to 36603f3 Compare August 28, 2025 05:23

zroubalik requested review from Copilot and dttung2905 September 16, 2025 20:06

Copilot AI reviewed Sep 16, 2025

View reviewed changes

patelvp force-pushed the add-partition-based-scaling-kafka-scaler branch from 36603f3 to 1e8412a Compare October 1, 2025 15:54

patelvp force-pushed the add-partition-based-scaling-kafka-scaler branch from 1e8412a to 315364c Compare October 1, 2025 16:02

wozniakjan approved these changes Oct 2, 2025

View reviewed changes

wozniakjan merged commit 64a3d8d into kedacore:main Oct 2, 2025
24 checks passed

Add support for partition based scaling on the kafka scaler #6558

Add support for partition based scaling on the kafka scaler #6558

Uh oh!

Conversation

patelvp commented Feb 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

JorTurFer commented Feb 20, 2025

Uh oh!

dttung2905 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zroubalik commented Feb 25, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patelvp commented Mar 3, 2025

Uh oh!

patelvp commented Mar 6, 2025

Uh oh!

dttung2905 commented Mar 8, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

utkuaydn commented Apr 15, 2025

Uh oh!

patelvp commented Apr 16, 2025

Uh oh!

rickbrouwer commented Jun 15, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patelvp commented Jun 20, 2025

Uh oh!

rickbrouwer commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patelvp commented Aug 20, 2025

Uh oh!

zroubalik left a comment

Choose a reason for hiding this comment

Uh oh!

rickbrouwer commented Aug 26, 2025

Uh oh!

rickbrouwer commented Aug 27, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patelvp commented Aug 27, 2025

Uh oh!

patelvp commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rickbrouwer commented Aug 28, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patelvp commented Aug 28, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zroubalik commented Sep 18, 2025

Uh oh!

dttung2905 commented Sep 18, 2025

Uh oh!

zroubalik commented Sep 30, 2025

Uh oh!

patelvp commented Sep 30, 2025

Uh oh!

patelvp commented Oct 1, 2025

Uh oh!

rickbrouwer commented Oct 2, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

patelvp commented Feb 20, 2025 •

edited

Loading

zroubalik commented Feb 25, 2025 •

edited by github-actions bot

Loading

dttung2905 commented Mar 8, 2025 •

edited by github-actions bot

Loading

rickbrouwer commented Jun 15, 2025 •

edited by github-actions bot

Loading

rickbrouwer commented Jun 20, 2025 •

edited

Loading

rickbrouwer commented Aug 27, 2025 •

edited by github-actions bot

Loading

patelvp commented Aug 28, 2025 •

edited

Loading

rickbrouwer commented Aug 28, 2025 •

edited by github-actions bot

Loading

rickbrouwer commented Oct 2, 2025 •

edited by github-actions bot

Loading