Auto instrumentation with bunny

[ruby4mag]

Hello Team,

I have a use case like below,

1. autoinstrumentation enabled rails server that publishes messages to a rmq queue . I see trace meta data in the message headers

{:content_type=>"application/octet-stream", :headers=>{"traceparent"=>"00-4bf8fc375a1a796cf8e9ac79f9e18fa0-f714722dfa4b3139-01"}, :delivery_mode=>2, :priority=>0, :tracer_receive_headers=>{"traceparent"=>"00-def04aabef1dff356414dd01fb96f05b-c429bb60b5b843e0-01"}}

2. A ruby bunny subscriber script, consumes messages from the queue in a loop and sends to another rails server. I have enabled auto instrumentation on this script as well.

The issue is, when I visualize traces in wavefront, I see the publish transaction as a separate trace and, consume and forward as a separate trace. Ideally, it should have been a single trace starting with publish span and continue with consume and forward spans.

It means bunny auto instrumentation is not getting/using the context information from the message headers and instead , creating a new trace.

Any lights on this please ?

[sokratisg]

@ruby4mag did you manage to find any solution to this issue? Been having the same behavior on X-Ray and couldn't figure out why it's happening. Although I haven't yet checked otel's debug logs, I am pretty sure lack of context propagation or respect of it is the root cause.

[arielvalentin]

The Bunny instrumentation adheres to the OTel messaging specification and uses Links instead of Child Spans in consumers. https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/messaging.md#batch-receiving

Links lead to new traces, which is why you're seeing a different trace is for your RabbitMQ spans.

In the end though, this is probably a limitation of your tracing provider, which likely does not know how to visualize links or how to correlate them.

https://opentelemetry.io/docs/reference/specification/trace/api/#specifying-links

[mugli]

@arielvalentin The OTEL AMQP instrumentation in the Node.js land does not do it like this and creates a child span on receiving messages instead of creating new trace and linking them: https://github.com/open-telemetry/opentelemetry-js-contrib/blob/4e18a46396eb2f06e86790dbbd68075c4c2dc83b/plugins/node/instrumentation-amqplib/src/amqplib.ts#L409-L432

Now I don't know who is doing it right or wrong.

The bunny instrumentation also does not propagate baggage or tracecontext at all when it creates the new trace when receiving messages. I believe this is a bug.

[arielvalentin]

@blumamir given that you worked on the AMQP instrumentation for Node.js I hope you may be able to provide some context here.

@mugli if the bunny instrumentation is missing baggage and trace context and you assert this is a bug please open an issue in the opentelemetry-ruby-contrib repo cc: @johanneswuerbach @fbogsany #566

[mugli]

I created an issue here: https://github.com/open-telemetry/opentelemetry-ruby-contrib/issues/523

[mugli]

@arielvalentin Additionally, here is a simplified reproduction of the problem: https://github.com/mugli/otel-ctx-prop-amqp-pubsub