Skip to content

feat: add OpenTelemetry span links to MQTT pub/sub#3187

Open
majiayu000 wants to merge 1 commit intogofr-dev:developmentfrom
majiayu000:feat/issue-3012-mqtt-span-links
Open

feat: add OpenTelemetry span links to MQTT pub/sub#3187
majiayu000 wants to merge 1 commit intogofr-dev:developmentfrom
majiayu000:feat/issue-3012-mqtt-span-links

Conversation

@majiayu000
Copy link

Pull Request Template

Description:

  • Fixes Add span links in MQTT #3012
  • Add OpenTelemetry span links to MQTT pub/sub, mirroring the Kafka (PR feat(kafka): add span links for pub/sub tracing #2952) and Google PubSub trace propagation patterns.
  • MQTT v3.1.1 does not support message-level headers, so trace context is propagated via payload wrapping: publisher injects W3C trace context into a JSON envelope, subscriber extracts it and creates span links to the producer span.
  • New file tracing.go provides headerCarrier (implementing propagation.TextMapCarrier), payload wrap/unwrap, and span creation helpers.
  • Backward compatible: non-wrapped payloads pass through unchanged.

Breaking Changes (if applicable):

  • None. Published payloads are now wrapped in a JSON envelope with trace context, but subscribers transparently unwrap them. Legacy consumers receiving wrapped messages will see the JSON envelope as the payload.

Additional Information:

  • Uses go.opentelemetry.io/otel and go.opentelemetry.io/otel/trace (already project dependencies).
  • Pattern mirrors pkg/gofr/datasource/pubsub/kafka/tracing.go and pkg/gofr/datasource/pubsub/google/tracing.go.

Checklist:

  • I have formatted my code using goimport and golangci-lint.
  • All new code is covered by unit tests.
  • This PR does not decrease the overall code coverage.
  • I have reviewed the code comments and documentation for clarity.

Thank you for your contribution!

Add trace context propagation to MQTT using payload wrapping,
since MQTT v3.1.1 has no message-level headers. Publisher injects
W3C trace context into a JSON envelope; subscriber extracts it
and creates span links to the producer span.

New file tracing.go provides headerCarrier, payload wrap/unwrap,
and span creation helpers mirroring the Kafka and Google PubSub
patterns already in GoFr.

Signed-off-by: majiayu000 <1835304752@qq.com>
@majiayu000 majiayu000 marked this pull request as ready for review March 22, 2026 06:46
@Umang01-hash
Copy link
Member

Hey @majiayu000 ! Thanks for the effort to make GoFr better. I also opened a PR #3101 to solve the same issue but the PR was closed as:

MQTT does support metadata that can help facilitate smoother span link additions. However, we should address this after upgrading MQTT or by identifying a simpler approach to achieve proper correlation.

At the moment, adding independent span links does not seem to serve the intended purpose effectively. Hence, I am closing this PR for now.

I request the author to please discuss the proposed approach on the issue ticket first. Once the approach is reviewed and approved, the PR can be reopened or a new PR can be created accordingly.

The core issue with both approaches comes down to MQTT protocol versions:

  • MQTT 3.1.1 (what we currently use) has no concept of message-level metadata/headers — so trace context has nowhere to live natively.
  • MQTT 5.0 introduced User Properties, which work exactly like Kafka headers or SQS message attributes — a clean key-value metadata map perfect for injecting traceparent.

Your payload-wrapping approach does achieve span correlation, but it has a significant downside: it changes the actual message content. Any subscriber not using GoFr (or an older GoFr version) will receive a JSON envelope like {"payload": "...", "traceparent": "..."} instead of the original data. That's a silent breaking change for mixed
environments.

The right path forward, as hinted by the maintainers, is to upgrade the MQTT driver to support MQTT 5.0, which would let us inject and extract trace context in User Properties — no payload modification needed, consistent with how Kafka/SQS/Google PubSub do it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add span links in MQTT

2 participants