OpenTelemetry Tracing API vs Tokio-Tracing API for Distributed Tracing

[cijothomas]

Background

The Rust ecosystem has two prominent tracing APIs: the OpenTelemetry Tracing API (Otel for short), delivered through the opentelemetry crate, and the Tokio tracing API, provided by the
tracing crate. The OTel Tracing API adheres to the OpenTelemetry specification, ensuring alignment with OpenTelemetry Tracing implementations in other languages like C++, Java etc. Conversely, the Tokio tracing ecosystem, which predates
OpenTelemetry, boasts widespread adoption, with many popular libraries already instrumented. The tracing-opentelemetry crate, maintained outside of OpenTelemetry repositories, act as a "bridge", enabling applications instrumented with tracing to work with OpenTelemetry.

The issue

The coexistence of the OTel Tracing API and Tokio-Tracing poses a dilemma, forcing end users to choose between two competing APIs. This situation complicates the decision-making process due to the absence of comprehensive
documentation comparing the two options. A significant concern is the lack of tested interoperability between the APIs, which can result in issues, especially in applications where different layers use different tracing APIs, potentially
leading to incomplete traces. This also impacts the log correlation scenarios as well.

A Comparison with OTel .NET

The OpenTelemetry .NET community encountered a similar challenge when the OTel Tracing API was introduced, as the .NET runtime library (shipped as the
DiagnosticSource package) already had a similar API in place. This issue was resolved through collaboration between OTel .NET maintainers and the .NET
runtime team, leading to the alignment of the .NET runtime's tracing API with the OTel specifications. This approach was later applied to the Metrics API as well. While the decision by OTel .NET to prioritize the .NET Runtime library's
API over its own for tracing/metrics has generally been successful, it has not been without its challenges. Despite declaring stability years ago, OTel .NET has yet to implement certain aspects of the OTel specification fully.

Although the outcomes in the .NET ecosystem might not directly forecast the success of similar efforts in Rust, they provide a valuable reference point.

Options for Consideration

1. Deprecate Tokio-Tracing: This approach would align Rust with the OpenTelemetry strategies adopted by other languages. However, considering the popularity and active maintenance of the tracing crate in the Rust ecosystem, this path has highest friction and is highly improbable.

2. Deprecate OTel Tracing: Promoting Tokio-Tracing as the standard could be a feasible option, albeit requiring comprehensive evaluation. This strategy would cause OTel Rust to deviate from its counterparts in other languages.
   Potential alignment of Tokio-Tracing with OTel Tracing specifications could mitigate this concern but necessitates groundwork to identify gaps and propose solutions. Tokio-Tracing maintainers have shown willingness to accommodate
   reasonable changes, pending a clear set of requirements. This option does not eliminate the OTel Tracing API completely, but it'll still remain to compensate for things missing from Tokio-Tracing - only those APIs which are overlapping/competing with Tokio-Tracing needs to be deprecated/removed.

3. Maintain Both APIs: This alternative emphasizes the importance of ensuring seamless interoperability between the two APIs, allowing users to choose based on preference or specific needs without compromising trace completeness. Achieving this goal requires significant effort to identify and bridge any existing gaps in the interoperability story. Users should be able freely chose between, without worrying about any broken traces.

4. Do nothing.: OTel Rust has some special accommodations done to help tracing crate (and vice-versa). We can just remove them, and let each crate follow their own destiny. (Highly undesirable state, just listed for completion)

Are there more options? Please let us know in the comments!

Current State

The Rust tracing ecosystem is at a critical juncture. Active discussions between the OTel Rust team and the Tracing Rust team are taking place, with updates and deliberations shared on Cloud Native
Slack. Interested individuals are encouraged to join the discussion on Slack (or right in this Github issue). All decisions and considerations will be posted on GitHub as well for wider visibility and to gather feedbacks.

Timeline

Resolving this issue is a prerequisite (though not the only one) for declaring the Tracing signal as GA (General Availability) for OTel Rust. Given the goal to achieve Tracing GA (alongside other milestones) soon, it's crucial that this issue is resolved promptly. A tentative deadline to reach a decision on the chosen path forward is set for April 30th, 2024, approximately 2 months from today.

Related issues

#1378 Tracing Propagation.
#1394 (comment)
Broken Trace example : #1690

[cijothomas]

Tagging @open-telemetry/rust-approvers
@jtescher as tracing-opentelemetry maintainer
@davidbarsky as tracing maintainer

[TommyCpp]

If we were to use tracing as the API. This is the deviation between existing tracing API and Otel tracing API

• - means no support today
• ? means no support but pontential way to implement it
• + means supported

Feature	Tokio Tracing
TracerProvider
Create TracerProvider	-
Get a Tracer	-
Get a Tracer with schema_url	-
Get a Tracer with scope attributes	-
Associate Tracer with InstrumentationScope	-
Safe for concurrent calls	+
Shutdown (SDK only required)	N/A
ForceFlush (SDK only required)	N/A
Trace / Context interaction
Get active Span	+
Set active Span	+
Tracer	Optional
Create a new Span	+
Documentation defines adding attributes at span creation as preferred	+
Get active Span	+
Mark Span active	+, same as tracing enter
SpanContext
IsValid	?, can add as event or metadata
IsRemote	?, can add as event or metadata
Conforms to the W3C TraceContext spec	-
Span	Optional
Create root span	+
Create with default parent (active span)	+
Create with parent from Context	+
No explicit parent Span/SpanContext allowed	+
SpanProcessor.OnStart receives parent Context	-
UpdateName	-
User-defined start timestamp	-
End	+, when span closes(not exit)
End with timestamp	-
IsRecording	?, can add attributes or metadata
IsRecording becomes false after End	?
Set status with StatusCode (Unset, Ok, Error)	?, can add as event or metadata
Safe for concurrent calls	+
events collection size limit	-
attribute collection size limit	-
links collection size limit	-
Span attributes	Optional
SetAttribute	+
Set order preserved	-
String type	+
Boolean type	+
Double floating-point type	+
Signed int64 type	+
Array of primitives (homogeneous)	+
null values documented as invalid/undefined	N/A
Unicode support for keys and string values	+
Span linking	Optional
Links can be recorded on span creation	?, can add as metadata
Links can be recorded after span creation	?, can add as metadata
Links order is preserved	?, depends on subscriber
Span events
AddEvent	+
Add order preserved	?, depends on subscriber
Safe for concurrent calls	+
Span exceptions
RecordException	?, can add as events w/ special flags
RecordException with extra parameters	?, can add as events w/ special flags, see tracing-error
Sampling	Optional
Allow samplers to modify tracestate	N/A
ShouldSample gets full parent Context	N/A
Sampler: JaegerRemoteSampler	N/A
New Span ID created also for non-recording Spans
IdGenerators	?, not supported directly but can add via subscribers
SpanLimits	-
Built-in SpanProcessors implement ForceFlush spec	N/A
Attribute Limits	-
Fetch InstrumentationScope from ReadableSpan	-

[hdost]

From the metrics perspective exemplars are also something to take into account.

[hdost]

As requested in the community meeting:
I am partial towards Option 2.
Specifically I don't think we'd eliminate the API surface as we're currently supporting basically all the needed features in the spec.

I would like say that we should probably try to see if it's not possible to improve the inter-compatibility as people will still try to use it directly.

Questions that are open from my perspective:

• What are the minimal set of features we feel like we'd want to maintain?
• Are the tokio team amenable to supporting such features?

We know that the tracing crate is widely used in the community, but so is the opentelemetry SDK, it's hard to necessarily see how many directly instrument with OpenTelemetry.

Update: I think really I'd be more 3 than 2. If we can promote inter-compatibility between the two then I think that's a greater win for the community at large. Because as I mentioned during the meeting we will still need to have "some" API anyway.

[ramosbugs]

I am partial towards Option 2.
Specifically I don't think we'd eliminate the API surface as we're currently supporting basically all the needed features in the spec.

As a heavy user of direct OpenTelemetry instrumentation (e.g., using SpanBuilder a lot, along with span events and span links) in the backend of my AWS Lambda-based web app, I'm nervous reading this. There are almost 800 calls to set_attribute alone in my codebase, and moving off of a deprecated API used this heavily would be a major undertaking.

Which interfaces specifically would be deprecated?

We know that the tracing crate is widely used in the community, but so is the opentelemetry SDK, it's hard to necessarily see how many directly instrument with OpenTelemetry.

I suspect a lot of other OpenTelemetry users are also doing so in private repositories, so I agree that it's hard to measure. I would caution against inferring much from these public GitHub usage stats.

[TommyCpp]

Which interfaces specifically would be deprecated?

We don't know exactly yet. The idea is to bridge the gap between the tracing and OTEL spec using custom API but if something tracing supports we want to deprecate in favor of that. But we are still debating ideas and nothing has been decided yet. Thus, any feedback is greatly appreciated!

[lalitb]

I vote for option 2, as there are challenges with other options:

• Option 1: Tokio-Tracing is deeply ingrained in the Rust ecosystem with wide adoption across many libraries and applications. It's practically not feasible to ask community to migrate from tokio-tracing to Otel tracing API.
• Option 3: Keeping OpenTelemetry API, while also supporting tracing-opentelemetry would increase maintenance effort., add confusion to the developers on which API to choose, and interop across both would be a challenge (as seen here)
• Option 4: This approach, while minimizing immediate effort, overlooks the broader implications for the Rust ecosystem's health and future growth.

Going with Option 2, we also need evaluation for introducing an extension API within OpenTelemetry. This is to effectively bridge the existing gaps between the OTel specifications and Tokio-Tracing's functionalities (e.g, Baggage support, Propagators).