End to End tracing with multiple Kafka Clusters and MirrorMaker2

[Fobhep]

Hi,
I have a question and I am not sure I understand it generally wrong, or if I am configuring something wrong.

tl;dr - MirrorMaker2 seems to change message headers that contain the traceparent id. is there a way to keep it from changing it?

I have two Kafka Strimzi Clusters: dc1 and dc2

I have a producer (quay.io/strimzi-examples/java-kafka-producer:latest) in dc1, a mirrormaker between dc1 and dc2 and a consumer in dc2 (quay.io/strimzi-examples/java-kafka-consumer:latest)

Basically like this:
Producer -> Dc1 -> Mirrormaker -> DC2 -> Consumer

Now if I simply add Jaeger to the game and enable tracing on producer and consumer I can see the flow very beautifully

I however would like to include MirrorMaker into the tracing-flow, so I can see wether something goes wrong while replicating.
So I am enabling tracing like this:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: dc1todc2
spec:
  version: 3.9.0
  replicas: 1
  connectCluster: "dc2" # Must be the target custer
  clusters:
  - alias: "dc1" # Source cluster
    bootstrapServers: http://dc1-kafka-bootstrap:9092
  - alias: "dc2" # Target cluster
    bootstrapServers: http://dc2-kafka-bootstrap:9092
    config:
      # -1 means it will use the default replication factor configured in the broker
      config.storage.replication.factor: -1
      offset.storage.replication.factor: -1
      status.storage.replication.factor: -1
  mirrors:
  - sourceCluster: "dc1"
    targetCluster: "dc2"
    sourceConnector:
      tasksMax: 1
      config:
        # -1 means it will use the default replication factor configured in the broker
        replication.factor: -1
        offset-syncs.topic.replication.factor: -1
        replication.policy.class: org.apache.kafka.connect.mirror.IdentityReplicationPolicy
        sync.topic.acls.enabled: "false"
        refresh.topics.interval.seconds: 600
        header.converter: org.apache.kafka.connect.converters.ByteArrayConverter  
    checkpointConnector:
      tasksMax: 1
      config:
        # -1 means it will use the default replication factor configured in the broker
        checkpoints.topic.replication.factor: -1
        replication.policy.class: org.apache.kafka.connect.mirror.IdentityReplicationPolicy
        header.converter: org.apache.kafka.connect.converters.ByteArrayConverter
        sync.group.offsets.enabled: "false"
        refresh.groups.interval.seconds: 600
    topicsPattern: "topic1"
  template:
    connectContainer:
      env:
        - name: OTEL_SERVICE_NAME
          value: mmdc1todc2
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://jaeger:4317"
  tracing:
    type: opentelemetry # (43)

If I now look at my traces I get not one complete end-to-end trace in Jaeger but only parts of the flow:

I get this part

Producer -> Dc1 -> Mirrormaker

and I get this part

Mirrormaker -> DC2 -> Consumer

This shows me nicely that MirrorMaker is doing something, but again it is not really helpful - since I would like to see the whole flow.

I am pretty sure that is due to the fact that MirrorMaker is not keeping the traceparent in the message header consistent, but changing it - one example:
( I am telling MirrorMaker2 to keep topic1 as is between DC1 and DC2 - same name, same config etc, so messages and headers in same offsets should be identical)

DC1- topic1 - partition0:  offset 5099
Headers: {"traceparent":"00-7cd30475c5daf237889e341832e23c27-f2a41a75615fb897-01"}
Value: Hello-World - 5796

DC2- topic1 -  partition0:  offset 5099
Headers: {"traceparent":"00-54d743972183ac523221576dc3cc39e6-4cd841e504145bf6-01"}
Value: Hello-World - 5796

Is there a way to stop MirrorMaker2 from changing the header and just reuse it?
Or is this a different problem?

[Fobhep]

I will add something I found - maybe that is correct.
Investigating further I found this thread: https://cloud-native.slack.com/archives/CMH3Q3SNP/p1708351090783599 and this open-telemetry/opentelemetry-java-instrumentation#6297 and if I get this correct:

The OTEL default producer interceptor will create a new trace-span when used and not keep the current one alive?

So in order to get this fixed - one would either need a different producer interceptor or add that feature to the OTEL default kafka prodcuer interceptor?
Nothing that can be done from MirrorMaker2 config-side

[katheris]

Hi @Fobhep, I'm not very familiar with OTEL, but from what you've shared I think you are correct that it's because Mirror Maker is consuming and then creating it's own producer, which is what is triggering the new trace-span to be used. So I agree that having an option for the OTEL default kafka producer interceptor to reuse an existing trace-span seems to me to be the right move.

[ppatierno]

@Fobhep yes you are right. As you saw I faced this issue in the past mainly because how the Kafka producer works (having a thread filling the buffer and another sending the message) so the context is kind of lost across threads. Anyway there is nothing we can do at Mirror Maker level, it's part of the OTEL producer interceptor.

[Fobhep]

@katheris @ppatierno thanks for your answers!
Just thinking out loud - would the Strimzi project be up for having a custom (and optionally enabled) Interceptor shipped inside that would solve that issue? I am asking since I might come into the situation where I have to find a solution somehow and that might include writing such an Interceptor ;)

[ppatierno]

@Fobhep I would be more for engaging with the open telemetry community and explaining the situation and maybe contributing there. If you start any discussion feel free to tag me so I can follow.

[Fobhep]

sorry for the late reply - yes that makes sense. Thanks I will keep it in mind :)