Make Sidekiq `propagation_style` able to be changed on a job-by-job basis

[hibachrach]

Right now, you can choose to have all jobs connected to their "producer" (enqueuing) spans via a link or via a standard parent-child span relationship (child) via the global propagation_style option. However, that level of granularity is not always sufficient for accurately representing a user-journey as a trace.

For example, let's say we have a web request where a user is uploading an document to convert it from one file type to another. This request handler

1. Enqueues a background job to perform that file conversion, and
2. Enqueues a separate background job for analytics processing

In terms of a user journey, we only actually care about (1) and not about (2). Ideally, I'd be able to connect the enqueue span to the file processing job performance span with a parent-child relationship and use a link for the other enqueue.

I'm open to implementing this functionality if such a feature would be welcome.

[arielvalentin]

Thanks for being willing to help with these here.

I think what we need are champions upstream to help with coming up with specification changes.

Right now there isn't anything that specifies behavior to transition between different styles of context propagation per message or per request.

We have a similar situation/issue with server spans. For example, the Golang instrumentation allows users to configure per request handler context propagation but there isn't anything in the specification that describes this or how it should be implemented.

[hibachrach]

I'm definitely new to the Otel documentation but I'm wondering if

1. this is something that is expected to be specified in the specification
2. one needs to be blocked by the other?

That said, I'm happy to work on both in parallel. I'm assuming being an upstream champion would be following this process?

[arielvalentin]

Language specific idioms are generally where we provide more flexibility.

I think it's in our best interest to advocate at the spec level for these kinds of things. Especially when they have to do with core concepts like context propagation.

The more we deviate from the spec and one off cases the more challenging it is for us to keep up with maintenance and provide a predictable experience for our users.

We need more real life use cases like yours to drive change so your voice will be a huge help for us to advocate for spec level changes!

[github-actions]

👋 This issue has been marked as stale because it has been open with no activity. You can: comment on the issue or remove the stale label to hold stale off for a while, add the keep label to hold stale off permanently, or do nothing. If you do nothing this issue will be closed eventually by the stale bot.

[hibachrach]

@arielvalentin as a temporary measure, how would you feel about making it possible for the configuration option to be a proc? I think that enables a lot more flexibility without having to deal with the spec but leaves room open for the spec in the future to address this more comprehensively?