OTEP: Stable by Default

[austinlparker]

Summary

This OTEP proposes that OpenTelemetry distributions enable only stable components by default, decouple instrumentation stability from semantic convention stability, and establish expanded stability criteria.

Key Proposals

• Stable by default: Distributions should only enable stable components; experimental requires explicit opt-in
• Decouple instrumentation/semconv stability: Allow instrumentation to stabilize independently when API surface is stable, even if semantic conventions are still experimental
• Expanded stability criteria: Stability means more than API compatibility—includes documentation, benchmarks, tested integrations
• Unified component metadata: Extend Collector's metadata.yaml pattern to instrumentation libraries

Motivation

Community feedback consistently identifies pain points that this OTEP addresses:

• Experimental features breaking production deployments
• Semantic convention changes destroying dashboards
• Instrumentation libraries stuck on pre-release due to experimental semconv dependencies
• "Batteries not included" defaults that overwhelm newcomers

Related

• Responds to Stability Proposal Announcement
• Related discussion: Feedback on Stability and Release Practices Proposal community#3098
• Builds on OTEPs 0143, 0232, 0227, 0152, 0155

Test plan

• Review by Governance Committee
• Review by affected SIGs (Collector, SDK maintainers)
• Community feedback period

[jsuereth]

I actually think the work detailed in this OTEP is large, sweeping and likely needs to be divided up.

I suggest you keep the goals of the OTEP in this, and set up workstreams / requirements that can be tackled in further OTEPs. I.e. This is bigger than one person or one "design".

E.g.

• Enabling experimental features - A workstream we can ask the configuraiton SIG to drive.
• Federated Schema and declaring stability of a schema independently of semantic conventions - You can give this to Weaver / Semconv Tooling SIG (@lmolkova has an OTEP already to continue making progress here)
• Distributions / Releasing - A workstream around defining what a distribution is, and gate-keeping its default features to those that are stable
• Profiling - A workstream around profiling overhead and providing features / capabilities / infrastructure to allow Maintainers to set up these tests if they don't have them and meet the requirements listed here.

That's just top of mind, but I think we could refactor this OTEP to call out each workstream and find owners for those.

[pellared]

PTAL @open-telemetry/dotnet-instrumentation-maintainers

[reyang]

Before going into the details, I have the same question as what @cijothomas mentioned here.

"This OTEP proposes that OpenTelemetry distributions enable only stable components by default" - what does this mean? If let's say Company XYZ released their distribution of OpenTelemetry Java SDK, and they included an unstable component, what would the OpenTelemetry community do? - Do we send attorneys to them?

[austinlparker]

Hi all - thank you for the robust discussion on this proposal, as well as the feedback from the discussion during the most recent GC/TC meeting. I've gone through and rewritten the entire thing to focus more on identifying specific workstreams and their potential owners in order to make this a more digestable set of changes. I will be resolving all comments on this PR; Please re-review at your leisure.

[austinlparker]

Before going into the details, I have the same question as what @cijothomas mentioned here.

"This OTEP proposes that OpenTelemetry distributions enable only stable components by default" - what does this mean? If let's say Company XYZ released their distribution of OpenTelemetry Java SDK, and they included an unstable component, what would the OpenTelemetry community do? - Do we send attorneys to them?

I do want to be specific to this point. We, as a project, obviously have no recourse against people who do not respect the project rules as written. That said -

• Providing a policy and having a project consensus that we will follow this policy means that we can work with CNCF to provide guarantees around that policy. For example, we could create a 'Certified OpenTelemetry Compatible' trademark (or something) and have adherence to the policy be a requirement of it. If someone used this improperly, the CNCF would be able to take legal action.
• Generally, open source works on handshakes and rep :) As long as our users and the community view us as a neutral arbiter and implementor of the standards we create, our word carries weight. If company XYZ uses unstable components (or in some way releases software that is not compliant with our guidance) we are under no compunction to advertise their software on our website, in the registry, etc. So while we cannot 'force' compliance, we can certainly withhold promotion of derivatives that do not respect our policies, processes, etc.

[mx-psi]

cc @open-telemetry/configuration-maintainers

[jack-berg]

The config SIG was initialized as a project with a particular scope, and as such is planning on shutting down upon stabilizing the specification. It can recharter / restart, but with different goals and potentially different people.

[mx-psi]

cc @open-telemetry/weaver-maintainers

[jsuereth]

Draft proposal to put all the pieces together for this workstream - #4906

[jsuereth]

I love our End User + Communications SIG - but is this the right owner?

I think examples of this are crafting the collector resiliency documentation, but the key questions to ask here involve core architectural decisions around architectures OTEL components support and making sure our releases fit into that cohesive whole.

In lieu of a better SIG, I'd suggest this belongs to the TC (today, by charter), and we should step up what we offer here.

[jsuereth]

I'd suggest each workstream is either adopted as the roadmap of an existing SIG (when that's the owner) or becomes a new project in the governance model, with a dedicated project owner, to make sure this succeeds.

[austinlparker]

@jsuereth just to clarify - the intent of stability here is as follows:

• an instrumentation library should declare stability if the instrumentation code itself is performant, stable, and safe for production use even if it depends on unstable semconv
• if an instrumentation library updates any of its api contracts (or telemetry contracts) in a breaking way then this should be considered a breaking change requiring a major version update

is this unclear from the current wording? i can add this more explicit guidance.

[austinlparker]

I'll be at the 2/10 spec call to discuss this OTEP further synchronously with maintainers.

The updates resolved my comment.

[jsuereth]

• a major version update

is this unclear from the current wording? i can add this more explicit guidance.

As stated in the spec all - I think we agree in high level, but the way this is worded appears to give license to something we do not want.

Stable instrumentation SHOULD NOT break output telemetry. This means, in your wording, you should not say the output telemetry is unstable. Just that the output telemetry is not in "global" semconv, but maintained locally instead.

if an instrumentation library updates any of its api contracts (or telemetry contracts) in a breaking way then this should be considered a breaking change requiring a major version update

I do not get this when reading the current proposal. This is my major beef with it, and I'd like to reword/rephrase so it's clear our stance on this.

[mx-psi]

This looks good to me, I would like to see approvals from mentioned SIGs before approving myself though

[jsuereth]

Thanks for fixing the wording here!

I forgot to come back and re-review, was working on specific proposals w/ @lmolkova for the Federeated Semconv piece, but this now looks good to me.

I think we'll need to kick of specific projects for areas which don't have SIG owners. Ideally, I think we should get a "single person who feels responsible" for a workstream, but we can sort out that detail later.

[trask]

Suggested change

	Community feedback consistently identifies several pain points. Experimental features break production deployments—users report configuration breaking between minor versions, silent failures in telemetry pipelines, and unexpected performance regressions that only appear at scale. As one practitioner noted: "The silent failure policy of OTEL makes flames shoot out of the top of my head."
	Community feedback consistently identifies several pain points. Experimental features break production deployments—users report configuration breaking between minor versions, silent failures in telemetry pipelines, and unexpected performance regressions that only appear at scale.

[trask]

Suggested change

Many instrumentation libraries are stuck on pre-release because they depend on experimental semantic conventions, even when the instrumentation API surface itself is mature and battle-tested. The "batteries not included" philosophy means users must assemble many components before achieving basic functionality. Documentation assumes expertise, and newcomers describe the experience as "overwhelming" with "no discoverability." Auto-instrumentation can add significant resource consumption that only becomes apparent at scale, with reports of "four times the CPU usage" compared to simpler alternatives. Users evaluating OpenTelemetry for production deployment need confidence in CVE response timelines, dependency hygiene, and supply chain security—areas where commitments are not well documented.

Many instrumentation libraries are stuck on pre-release because they depend on experimental semantic conventions, even when the instrumentation API surface itself is mature and battle-tested. The "batteries not included" philosophy means users must assemble many components before achieving basic functionality. Documentation assumes expertise, and newcomers describe the experience as "overwhelming" with "no discoverability." Auto-instrumentation can add significant resource consumption that only becomes apparent at scale. Users evaluating OpenTelemetry for production deployment need confidence in CVE response timelines, dependency hygiene, and supply chain security—areas where commitments are not well documented.

[trask]

These all stem from the same problem: OpenTelemetry's default configuration prioritizes feature completeness over production readiness.

this seems a bit of an oversimplification given some of the examples above

[trask]

Configuration SIG doesn't exist anymore:

• update readme to remove configuration working group community#3297

[nrcventura]

However, once an instrumentation library stabilizes, any breaking change to its telemetry output must be treated as a breaking change requiring a major version bump.

This seems like an unreasonable burden to place on things like auto instrumentation. Consider the example where an http client library is directly instrumented using OpenTelemetry APIs, and it is using the currently stable semantic conventions for http client calls. All auto instrumentation needs to do to enable capturing that telemetry, is to subscribe to that telemetry (ActivitySource or Meter in dotnet for example). The instrumentation version is directly coupled to the version of the http client library, and completely outside the control of auto instrumentation.

• Does this mean that there is an expectation that auto instrumentation implementations need to perform proactive testing to detect changes in the telemetry output for new library versions?
• Does auto instrumentation need a new major version whenever we want to support a new major version of 3rd party library that is natively instrumented?
• Will library authors consistently do a major version bump if the telemetry signal changes?
• Do we need something in this proposal specifically for auto instrumentation to call out how default instrumentations need to be managed?