Performance powered by OTel

[sl0thentr0py]

@krainboltgreene so there are several things that would need to be done to get performance tracing working with sentry-elixir. This SDK was so far handled by the community so it is not on par with what we call our Unified API for making common abstractions across all Sentry SDKs.

I will make 2 lists below, one of the bare minimum that would lead to quick opentelemetry based performance and the other more 'ideal' list where we make the SDK feature compatible in terms of performance with other SDKs.

Required

• Generalize the Envelope abstraction to also be able to send items of type transaction.
• Add Transaction and Span abstractions.
• Add a SpanProcessor and Propagator to hook into opentelemetry.

Optional

• The elixir SDK is further also missing Hub (for concurrency) and Scope abstractions but technically they can be ignored if a quick path to opentelemetry support is desired.

I will make a new issue out of this to track it in sentry-elixir. Unfortunately, this is a non-trivial amount of development work, so I can't give you very clear cut instructions on how you can contribute, but feel free to try stuff out and make a PR if you're interested and we can collaborate.

If there is sufficient interest from the community, we can also potentially prioritize me working on this as well next quarter.

Originally posted by @sl0thentr0py in getsentry/sentry#40712 (reply in thread)

## Other notes
- [ ] @smeubank Make sure users understand how to upgrade to AM3 plans, in order to unlock this functionality. Maybe a note in the docs can already be enough and point users in the right direction on how to self-serve

[tsloughter]

What does "Hub" for concurrency in Elixir look like? It looks to be a context store? Like how OpenTelemetry we use the process dictionary?

Does this mean Sentry does no intend to adopt the OpenTelemetry API?

Also, what is "opentelemetry based performance" mean here? Based on what I read of the SpanProcessor model it doesn't read like you'd be relying on the Otel SDK for span operations, so if you found performance better with OpenTelemetry I wouldn't be sure thats the case when combined with the SpanProcessor. But I don't yet fully understand what it is doing, so I may be wrong there :)

[sl0thentr0py]

What does "Hub" for concurrency in Elixir look like? It looks to be a context store? Like how OpenTelemetry we use the process dictionary?

something like that yea, basically the Hub needs to be cloned per concurrency unit, so we'll need to spec out what that looks like for elixir.

Does this mean Sentry does no intend to adopt the OpenTelemetry API?

not directly, we have our own Tracing model and Ingestion so we will only support OpenTelemetry indirectly via the SpanProcessor pathway.

it doesn't read like you'd be relying on the Otel SDK for span operations

we'd rely on the Otel SDK for instrumenting and recording spans but we need to convert them to the Sentry model to be able to ingest and store them on our side, this is what the SpanProcessor would do.

cc @smeubank for high-level product design discussion ^

[tsloughter]

Now I see, so the SpanProcessor updates a global store of SpanId->SentrySpan and then OnEnd will update that based on the finished OpenTelemetry Span.

[thbar]

If there is sufficient interest from the community, we can also potentially prioritize me working on this as well next quarter.

I'm not sure how to provide feedback on that, but we (at https://transport.data.gouv.fr) would be very interested to see performance monitoring supported for Elixir.

[josevalim]

For what is worth, the Ruby one seems to be done via OpenTelemetry: https://docs.sentry.io/platforms/ruby/performance/instrumentation/opentelemetry/ ?

[tsloughter]

@josevalim I think it has both. And the OpenTelemetry option is awkward to implement -- uses a span processor to basically do a parallel tracking of spans. I don't know that there will be another option other than the processor though as long as Sentry requires the implementation to create Transactions.

[hkrutzer]

There is also a JS implementation using OpenTelemetry, indeed using a span processor.

[sl0thentr0py]

as long as Sentry requires the implementation to create Transactions.

@tsloughter @josevalim we have an ongoing project to move away from our Transaction model gradually on the ingestion side, will keep this thread updated when we ship something production ready.