Open
Description
The instrumentation has not been compatible since 0.15.0. The monkey patches do not honor the current method signature of the Producer class and ends up reverting it back to earlier versions.
It also looks like we can no longer instrument each_batch starting in v0.20.0:
1) Error:
OpenTelemetry::Instrumentation::Rdkafka::Patches::Consumer::#each_batch#test_0001_traces each_batch call:
NotImplementedError: `each_batch` has been removed due to data consistency concerns.
This method was removed because it did not properly handle partition reassignments,
which could lead to processing messages from partitions that were no longer owned
by this consumer, resulting in duplicate message processing and data inconsistencies.
Recommended alternatives:
1. Implement your own batching logic using rebalance callbacks to properly handle
partition revocations and ensure message processing correctness.
2. Use a high-level batching library that supports proper partition reassignment
handling out of the box (such as the Karafka framework).
/opt/hostedtoolcache/Ruby/3.3.6/x64/lib/ruby/gems/3.3.0/gems/rdkafka-0.20.0/lib/rdkafka/consumer.rb:614:in `each_batch'
lib/opentelemetry/instrumentation/rdkafka/patches/consumer.rb:46:in `each_batch'
test/opentelemetry/instrumentation/rdkafka/patches/consumer_test.rb:196:in `block (3 levels) in <top (required)>'
7 runs, 30 assertions, 0 failures, 1 errors, 0 skips
rake aborted!
Command failed with status (1)
/opt/hostedtoolcache/Ruby/3.3.6/x64/bin/bundle:25:in `load'
/opt/hostedtoolcache/Ruby/3.3.6/x64/bin/bundle:25:in `<main>'
Tasks: TOP => test
(See full trace by running task with --trace)
We must ensure that instrumentations do not negatively impact gems or introduce regressions in functionality.
Metadata
Assignees
Labels
No labels