KEDA ScaledObject with minReplicaCount: 0 and Kafka Consumer Groups

[aryabharat]

Query:

When using KEDA to create a ScaledObject with minReplicaCount: 0, issues arise with Kafka consumers. If the Kafka topic has no messages or the retention period lapses, Kafka marks the consumer group as dead. Consequently, when a producer sends a new message, KEDA fails to scale the consumer from 0 to 1 because it cannot find the consumer group. Is there any kind of heartbeat mechanism for Kafka consumers in KEDA to prevent this issue.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: <name>
  
spec:
  minReplicaCount: 0
  maxReplicaCount: 18
  pollingInterval: 10
  scaleTargetRef:
    name: <name>
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka-broker.svc.cluster.local:9092
        consumerGroup: CONSUMER_GROUP_ID
        lagThreshold: 1000
        topic: <topic>
        offsetResetPolicy: 'latest'

Key Points:

Configuration Issue:

• After the consumer group is marked dead, a new message sent to the topic does not trigger KEDA to scale the consumer pods up from 0.
• KEDA cannot locate the dead consumer group, preventing it from scaling the necessary consumers.

[zroubalik]

How would you refresh the consumer group? Any ideas what can be done?

[aryabharat]

KEDA can provide different way to check that do we need to scale the consumer or not.

[aryabharat]

@zroubalik I can see that KEDA has addressed this issues in the doc, but what does it means

multiple triggers where one supplies topic to ensure lag for that topic will always be detected;

how we can create a custom trigger in KEDA?

https://keda.sh/docs/2.16/scalers/apache-kafka/