Skip to content

Conversation

@donovanbai-dd
Copy link

This PR speeds up consumer group metric collection, which can be very slow for large kafka clusters.

changes:

  1. Add --group.workers (default is 100) and --group.metrics.timeout (default is 5m). Instead of collecting metrics for each consumer group serially for each broker, use a separate goroutine for each group.
  2. For each consumer group, make a copy of the broker object before calling FetchOffset since each broker object does not support concurrent requests.
  3. Remove an unnecessary mutex lock that was happening for each partition of each topic.
  4. Add some debug logging for execution time details

Results on a test cluster
Before: 2 minutes to collect consumer group metrics
After: 14 seconds to collect consumer group metrics

module github.com/danielqsj/kafka_exporter

go 1.24
go 1.24.0
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@danielqsj
Copy link
Owner

lgtm, please fix the golang lint issue @donovanbai-dd https://github.com/danielqsj/kafka_exporter/actions/runs/20580739843/job/59777223997?pr=506

@donovanbai-dd
Copy link
Author

lgtm, please fix the golang lint issue @donovanbai-dd https://github.com/danielqsj/kafka_exporter/actions/runs/20580739843/job/59777223997?pr=506

it's fixed now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants