High CPU usage when getting last offset in topic

[nachtneblovaK]

Description

We have a project using apache kafka which consumes messages. After each consumption we use AdminClient to get last added offset in topic - we need it for metrics to watch how many messages are still in queue.

Code for getting last offset:

CPU Usage

Steps to reproduce

Call AdminClient.GetWatermarkOffsets while consuming message.

Checklist

Please provide the following information:

• Confluent.Kafka nuget version:
  1.0.0-beta3
• Client configuration:
  {
  topic.metadata.refresh.interval.ms: 30000,
  socket.blocking.max.ms: 1
  queue.buffering.max.ms: 1
  receive.message.max.bytes: 104857300,
  group.id: "inspireCloud",
  enable.auto.commit: false,
  offset.store.method: "none"
  }
• Operating system:
  CentOs 7.5.1804

[mhowlett]

hi @nachtneblovaK - can you provide a complete minimal example demonstrating the problem? then i'll try to replicate.

[nachtneblovaK]

Hi @mhowlett, I attached simple project which calls AdminClient.GetWatermarkOffsets every second.In a yip you can also find Dockerfile and script to build it (build-docker-file.bat). We run it on CentOs 7.5.1804.

KafkaCPUTest.zip

[geofflamrock]

Could this potentially be connected to confluentinc/librdkafka#2108 ? There seems to be a few high cpu usage issues that were referenced to this, and the rd_kafka_q_pop_serve method is listed in your perf trace and this seemed to be the issue in the fixed issue.