End to End tracing with multiple Kafka Clusters and MirrorMaker2

[Fobhep]

Hi,
I have a question and I am not sure I understand it generally wrong, or if I am configuring something wrong.

tl;dr - MirrorMaker2 seems to change message headers that contain the traceparent id. is there a way to keep it from changing it?

I have two Kafka Strimzi Clusters: dc1 and dc2

I have a producer (quay.io/strimzi-examples/java-kafka-producer:latest) in dc1, a mirrormaker between dc1 and dc2 and a consumer in dc2 (quay.io/strimzi-examples/java-kafka-consumer:latest)

Basically like this:
Producer -> Dc1 -> Mirrormaker -> DC2 -> Consumer

Now if I simply add Jaeger to the game and enable tracing on producer and consumer I can see the flow very beautifully

I however would like to include MirrorMaker into the tracing-flow, so I can see wether something goes wrong while replicating.
So I am enabling tracing like this:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: dc1todc2
spec:
  version: 3.9.0
  replicas: 1
  connectCluster: "dc2" # Must be the target custer
  clusters:
  - alias: "dc1" # Source cluster
    bootstrapServers: http://dc1-kafka-bootstrap:9092
  - alias: "dc2" # Target cluster
    bootstrapServers: http://dc2-kafka-bootstrap:9092
    config:
      # -1 means it will use the default replication factor configured in the broker
      config.storage.replication.factor: -1
      offset.storage.replication.factor: -1
      status.storage.replication.factor: -1
  mirrors:
  - sourceCluster: "dc1"
    targetCluster: "dc2"
    sourceConnector:
      tasksMax: 1
      config:
        # -1 means it will use the default replication factor configured in the broker
        replication.factor: -1
        offset-syncs.topic.replication.factor: -1
        replication.policy.class: org.apache.kafka.connect.mirror.IdentityReplicationPolicy
        sync.topic.acls.enabled: "false"
        refresh.topics.interval.seconds: 600
        header.converter: org.apache.kafka.connect.converters.ByteArrayConverter  
    checkpointConnector:
      tasksMax: 1
      config:
        # -1 means it will use the default replication factor configured in the broker
        checkpoints.topic.replication.factor: -1
        replication.policy.class: org.apache.kafka.connect.mirror.IdentityReplicationPolicy
        header.converter: org.apache.kafka.connect.converters.ByteArrayConverter
        sync.group.offsets.enabled: "false"
        refresh.groups.interval.seconds: 600
    topicsPattern: "topic1"
  template:
    connectContainer:
      env:
        - name: OTEL_SERVICE_NAME
          value: mmdc1todc2
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://jaeger:4317"
  tracing:
    type: opentelemetry # (43)

If I now look at my traces I get not one complete end-to-end trace in Jaeger but only parts of the flow:

I get this part

Producer -> Dc1 -> Mirrormaker

and I get this part

Mirrormaker -> DC2 -> Consumer

This shows me nicely that MirrorMaker is doing something, but again it is not really helpful - since I would like to see the whole flow.

I am pretty sure that is due to the fact that MirrorMaker is not keeping the traceparent in the message header consistent, but changing it - one example:
( I am telling MirrorMaker2 to keep topic1 as is between DC1 and DC2 - same name, same config etc, so messages and headers in same offsets should be identical)

DC1- topic1 - partition0:  offset 5099
Headers: {"traceparent":"00-7cd30475c5daf237889e341832e23c27-f2a41a75615fb897-01"}
Value: Hello-World - 5796

DC2- topic1 -  partition0:  offset 5099
Headers: {"traceparent":"00-54d743972183ac523221576dc3cc39e6-4cd841e504145bf6-01"}
Value: Hello-World - 5796

Is there a way to stop MirrorMaker2 from changing the header and just reuse it?
Or is this a different problem?

[Fobhep]

I will add something I found - maybe that is correct.
Investigating further I found this thread: https://cloud-native.slack.com/archives/CMH3Q3SNP/p1708351090783599 and this open-telemetry/opentelemetry-java-instrumentation#6297 and if I get this correct:

The OTEL default producer interceptor will create a new trace-span when used and not keep the current one alive?

So in order to get this fixed - one would either need a different producer interceptor or add that feature to the OTEL default kafka prodcuer interceptor?
Nothing that can be done from MirrorMaker2 config-side