Message consumption error: PollExceeded (Local: Maximum application poll interval (max.poll.interval.ms) exceeded)

[kpapadatos]

We are using KDI with Azure Event Hubs with Kafka and we are regularly getting this:

2024-07-30T18:08:42 [INFO] - kafka_delta_ingest: PRE_REBALANCE - Revoke
2024-07-30T18:08:42 [DEBUG] - kafka_delta_ingest: POST_REBALANCE - Revoke
2024-07-30T18:08:42 [ERROR] - kafka_delta_ingest: Ingest service exited with error Kafka { source: KafkaError (Message consumption error: PollExceeded (Local: Maximum application poll interval (max.poll.interval.ms) exceeded)) }
thread 'main' panicked at src/main.rs:185:14:
called `Result::unwrap()` on an `Err` value: Kafka { source: KafkaError (Message consumption error: PollExceeded (Local: Maximum application poll interval (max.poll.interval.ms) exceeded)) }
stack backtrace:
   0:     0x563bde15ddaf - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h1e1a1972118942ad
   1:     0x563bdc5b1e0b - core::fmt::write::hc090a2ffd6b28c4a
   2:     0x563bde126182 - std::io::Write::write_fmt::h8898bac6ff039a23
   3:     0x563bde15f899 - std::sys_common::backtrace::print::ha96650907276675e
   4:     0x563bde15f0be - std::panicking::default_hook::{{closure}}::h215c2a0a8346e0e0
   5:     0x563bde160333 - std::panicking::rust_panic_with_hook::hac8bdceee1e4fe2c
   6:     0x563bde15fc02 - std::panicking::begin_panic_handler::{{closure}}::h00d785e82757ce3c
   7:     0x563bde15fb59 - std::sys_common::backtrace::__rust_end_short_backtrace::h1628d957bcd06996
   8:     0x563bde15fb46 - rust_begin_unwind
   9:     0x563bdbbdca02 - core::panicking::panic_fmt::hdc63834ffaaefae5
  10:     0x563bdbbdce55 - core::result::unwrap_failed::h82b551e0ff2b2176
  11:     0x563bdda7ea3c - tokio::runtime::context::runtime::enter_runtime::h6708b0972b2db409
  12:     0x563bdda67359 - kafka_delta_ingest::main::h70fa0c80617658b2
  13:     0x563bdda6d4e3 - std::sys_common::backtrace::__rust_begin_short_backtrace::h0da8ae65fe5dba64
  14:     0x563bdda6659f - main
  15:     0x7f600723224a - <unknown>
  16:     0x7f6007232305 - __libc_start_main
  17:     0x563bdbc99871 - _start
  18:                0x0 - <unknown>

[kpapadatos]

I had to add the following options (ref. https://github.com/Azure/azure-event-hubs-for-kafka/blob/master/CONFIGURATION.md):

-K
max.poll.interval.ms=300000
-K
session.timeout.ms=30000
-K
heartbeat.interval.ms=3000

[kpapadatos]

I had to add the following options (ref. https://github.com/Azure/azure-event-hubs-for-kafka/blob/master/CONFIGURATION.md):

-K
max.poll.interval.ms=300000
-K
session.timeout.ms=30000
-K
heartbeat.interval.ms=3000

It happened again a few days later, even with the above configuration.

2024-08-14T12:44:35 [DEBUG] - kafka_delta_ingest: Could not send to statsd Connection refused (os error 111)
2024-08-14T12:44:35 [INFO] - kafka_delta_ingest: Delta version 27890 completed in 345580 milliseconds.
2024-08-14T12:44:35 [DEBUG] - kafka_delta_ingest: running the runloop
2024-08-14T12:44:35 [DEBUG] - kafka_delta_ingest: Could not send to statsd Connection refused (os error 111)
2024-08-14T12:44:35 [DEBUG] - kafka_delta_ingest: Could not send to statsd Connection refused (os error 111)
2024-08-14T12:44:35 [DEBUG] - kafka_delta_ingest: Async metrics receive loop terminated: receiving on an empty and disconnected channel
2024-08-14T12:44:35 [INFO] - kafka_delta_ingest: PRE_REBALANCE - Revoke
2024-08-14T12:44:35 [DEBUG] - kafka_delta_ingest: POST_REBALANCE - Revoke
2024-08-14T12:44:35 [ERROR] - kafka_delta_ingest: Ingest service exited with error Kafka { source: KafkaError (Message consumption error: PollExceeded (Local: Maximum application poll interval (max.poll.interval.ms) exceeded)) }
thread 'main' panicked at src/main.rs:185:14:
called `Result::unwrap()` on an `Err` value: Kafka { source: KafkaError (Message consumption error: PollExceeded (Local: Maximum application poll interval (max.poll.interval.ms) exceeded)) }
stack backtrace:
   0:     0x55568001920f - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h1e1a1972118942ad
   1:     0x55567e3dc54b - core::fmt::write::hc090a2ffd6b28c4a
   2:     0x55567ffe1d72 - std::io::Write::write_fmt::h8898bac6ff039a23
   3:     0x55568001acf9 - std::sys_common::backtrace::print::ha96650907276675e
   4:     0x55568001a51e - std::panicking::default_hook::{{closure}}::h215c2a0a8346e0e0
   5:     0x55568001b793 - std::panicking::rust_panic_with_hook::hac8bdceee1e4fe2c
   6:     0x55568001b062 - std::panicking::begin_panic_handler::{{closure}}::h00d785e82757ce3c
   7:     0x55568001afb9 - std::sys_common::backtrace::__rust_end_short_backtrace::h1628d957bcd06996
   8:     0x55568001afa6 - rust_begin_unwind
   9:     0x55567d9db7d2 - core::panicking::panic_fmt::hdc63834ffaaefae5
  10:     0x55567d9dbc25 - core::result::unwrap_failed::h82b551e0ff2b2176
  11:     0x55567f8e503d - tokio::runtime::context::runtime::enter_runtime::h72551ded11a58576
  12:     0x55567f915e19 - kafka_delta_ingest::main::h9fa5ceaf23b36ce9
  13:     0x55567f93d703 - std::sys_common::backtrace::__rust_begin_short_backtrace::hcd848ffc8b624859
  14:     0x55567f91505f - main
  15:     0x7f8d2998f24a - <unknown>
  16:     0x7f8d2998f305 - __libc_start_main
  17:     0x55567daa4d81 - _start
  18:                0x0 - <unknown>

[BoedieJB]

We are facing the same at my project, only for one particular kafka topic. We face this issue after upgrading our image to the latest available. Tinkering with these settings now.