kafka consumer timeout errors occur on the query node

[pipi-olo]

Hello, I always appreciate your help. 👍👍👍👍
I am using Helm Chart version 4.2.19 with Milvus App Version 2.4.23.

Kafka subscription timeout logs are appearing on all Query Nodes.
[WARN] [kafka/kafka_consumer.go:139] ["consume msg failed"] [topic=by-dev-rootcoord-dml_7] [groupID=querynode-491-by-dev-rootcoord-dml_7_455435630134348838v2-true] [error="Local: Timed out"]
This is happening with all topics and all groupIds.

In Kafka 3.x, There is no result for the following command.
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

However, I am inserting entities into the collection in real-time. (in attu)

Q. How did this happen?
Q. Can I control the kafka consumer timeout?

[yhmo]

The error is thrown from here:

milvus/pkg/mq/msgstream/mqwrapper/kafka/kafka_consumer.go

Line 139 in b119ac5

log.Warn("consume msg failed", zap.String("topic", kc.topic), zap.String("groupID", kc.groupID), zap.Error(err))

kc.c.ReadMessage(readTimeout) ------------------- this line is to invoke the thirdparty kafka client confluent-kafka-go/kafka/consumer to read messages from kafka: https://github.com/confluentinc/confluent-kafka-go/blob/738a8b577c6f6b0cb5cb1bc4e8e8b50dc530d2f6/kafka/consumer.go#L485

The "readTimeout" value is configured in the milvus.yaml, default value is 10

kafka:
  brokerList: localhost:9092
  ......
  readTimeout: 10

Double-check the "readTimeout" in your milvus.yaml, ensure its value is greater than 1

The thirdparty kafka client returns the error "Local: Timed out", related issues: confluentinc/confluent-kafka-go#842
I believe something is wrong in the kafka cluster.

[pipi-olo]

I am using Kafka 3.8 installed via the Bitnami helm chart.
I use the externalKafka of Milvus helm chart.
In the milvus.yaml file of the query node, the readTimeout section is commented out.

In this case, which value does the readTimeout follow?

[pipi-olo]

If I set the readTimeout value in user.yaml, will that option still apply even if I use externalKafka?

[yhmo]

The default value of kafka.readTimeout is 10. I think it indeed could not access the kafka service.

Restart the milvus, and use this script to export the log: https://github.com/milvus-io/milvus/tree/master/deployments/export-log