[OTEP] Federate Semantic Conventions

[jsuereth]

DO NOT MERGE

This is a draft proposal to federate the semantic-conventions repository to allow for faster innovation and evolution across the ecosystem.

• We retain clear ownership of semantic convention domains
• We retain clear policy and tooling support for instrumentation authors to provide comprehensive documentation, stability guarantees and integration test assurance of the signals the produce.
• We provide a clear "shared" location for x-cutting attributes and definitions.
• We provide a path for federated semantic conventions to promote to core, if desired.
• We expand for non-opentelemetry sources of conventions and standards to advertise their instrumentation and leverage the same tooling as open-telemetry.

[lmolkova]

so that there is no chance of having conflicts / vendoring in in incompatible manner / etc?

I like it!

[lmolkova]

this is awesome!

[lmolkova]

nit: maybe explain what policies are (link to some section/doc), I imagine it's not a commonly known concept

[lmolkova]

nit: it's the same on as the last one? (policies/check/naming)

[lmolkova]

nit:

I believe we now have the opt-in mechanism that's slightly different than picking a schema url, it was just introduced in declarative config: https://github.com/open-telemetry/opentelemetry-configuration/blob/4351ebd2805746d047a23588f8d3abe89f40f79f/snippets/ExperimentalInstrumentation_kitchen_sink.yaml#L52

E.g.

instrumentation:
  general:
    rpc:
      semconv:
        version: 1
        experimental: false
        dual_emit: true

[lmolkova]

nit: it's not part of weaver.yaml now, right? do we want to make it?

Or should we ship GH actions and/or have some shareable workflows that repos would reuse or copy-paste?

[lmolkova]

Do we want to allow stable (federated) registry to depend on unstable parts of otel conventions?

I think we can go either way.
If we don't allow it: they can take dependency on stable parts only and vendor in unstable parts if necessary.
If we allow it, they should be able to communicate breaking changes in unstable core via semver.

Either way, we need some forcing factors for them to

• update to newer underlying core - it will be hard regardless
• bring common concepts to core conventions

I'd rather not allow it at least initially and encourage to vendor in experimental stuff.

[lmolkova]

this would probably be the forcing factor to update federated registries I mentined in prev comment.
If library A depends on semconv core v1.40 and library B on semconv core v1.140 they probably can't coherently work in the same distro.

[lmolkova]

nit: would we separate HTTP from core conventions? Probably not until it needs v2

[lmolkova]

This is awesome!

I also like the year as a major version - we should create expectation of some breaking changes on a predictable cadence

[mikeldking]

This is date ver no? This is not semver.

[lmolkova]

we should maintain a list of federated conventions in semconv (or somewhere) for discoverability. And using it we can even have a weekly check that validates all federated registries together to find conflicts or tests them individually against latest core. If some collisions are found, we could automatically create issues and notify maintainers about conflicts.

It would also be a good forcing factor for conventions to stay in sync with core when they can.

[jsuereth]

Added

[lmolkova]

[update] I see you have it addressed later

assuming it's not required, what would be the motivation to even start bringing semconv to core?

I can imagine these:

• offload maintenance of a stable thing to core repo
• need to align between different federated repos (e.g. align db server with db client)
• need to align compatibility for platform release

But unless something starts to break, I don't see people being too interested in this work. And I think it's usually fine, but it's likely that we'll grow into 5 different versions of some conventions across 5 different repos.

We should probably require sigs like GenAI to be language-agnostic to avoid it.

[jsuereth]

Absolutely agree. I think that's the inevitability of federating and we'll need to balance cost / benefit here. I expect we'll have two types of fedaration:

• jvm / go runtime like things where it's really a specific technology that needs to be addressed. Unlikely we'll need to merge this back in ever.
• GenAi, db, http type things where they need to be cross language and cross cutting. These main want to come back into core, so their attributes can be re-used in further federated registries without as much dependnecy hell, but do not need to.

[mikeldking]

How does this propagate to natively instrumented libraries and what incentive do they have to move to newer conventions? There always is a transform layer running?

[mikeldking]

are conventions in a particular version all uniformly considered "stable"?

[mikeldking]

what are the rules of semver here. Like dropping an attribute is "breaking" - what if it was optional?

[mikeldking]

What's the acceptance criteria for "tested together", what if an instrumentation fails, is it removed from release?

[mikeldking]

What if end-users want to opt out of expensive / PII risking conventions. Do these still land in core? What is the final say for something deserving to be in core. I'm unfamiliar here.

[jsuereth]

We allow expensive / PII in core, with opt_in today and there's guidance around it. I think this is an orthogonal concern to this proposal, but an important issue that needs a solution across various components of OpenTelemetry.

For this proposal, we continue with the current behavior we have.

[mikeldking]

this is a good example, but also centralized with the maintainers of Java. So it seems like a simpler example? It's harder to do this when there is no central vendor.

[mikeldking]

where to these actions run - on OTEL repos only? How does it run on native libraries.

[jsuereth]

So we only "enforce" this structure for OTEL-owned repositories/conventions.

For native libraries, we provide open-telemetry/weaver as a tool they can use to participate in this federation, but we do not enforce any structure / policy unless they are planning to move their definitions back into the OpenTelemetry project.

[mikeldking]

what if the federated conventions need the unstable conventions for it to make sense - example being session / threading of agents.

[jsuereth]

Then the federated convention would be marked "unstable" as well. Basically you can't declare a "child" convention stable unless the "parent" is also stable.

This doesn't preclude usage, just stability declaration.

[mikeldking]

dual_emit prioritizes which verion?

[jsuereth]

In OpenTelemetry - dual_emit will fire both conventions, i.e. you'll get both attributes, multiple metrics, etc.

@trask or @lmolkova may be able to speak more to how that works in practice.

[mikeldking]

which users are we talking about. The end-user?

[jsuereth]

Yes

[mikeldking]

this is not so simple. Instrumentation is dependent on now two versions, the library it instruments AND the semcov that it tracks. What if the instrumentation needs to move a major version for library changes, then subsequently needs to backfill support for an older version. Now there is no tenable way to track both semcov breaking changes with library breaking changes.

[jsuereth]

I think I intend this to mean the opposite, or at least I don't see them as coupled.

• If you make a breaking change in semconv, you need a major version bump on your library.
• You may need major version bump on your library for other reasons, you do NOT need to bump semconv major version for this.

The key here is folks understand they need to do the first, and that the output signals of an instrumentation library are part of its stability.

Regarding tracking both semconv breaking + library breaking changes - I wanted to keep this simpler where you mostly just pay attention to library major version bumps, and then schema_url and backends can help you handle incompatibilities / conversions downstream.

[mikeldking]

What if the library is unstable?

[jsuereth]

We would allow semconv to be marked as stable prior to libraries being fully stable. This is more to prevent the opposite in otel - libraries which are production-ready NOT declaring stability, because of a looming threat of semconv changes.

[mikeldking]

the problem here is "cost" - the cost to one OTLP backend might be significant - it dictates a business deal being made in a quarter vs losing to a competitor that doesn't comply to Otel at all. This puts vendors that are all in on OTEL at risk as their upstream signals are being dictated by parties that have no actual financial overhead or incentives to move the standards forward.

[mikeldking]

how can we say that backends will use these agent-aided transformations? That seems like something that certain vendors such as companies that have privileged foundation model access a competitive advantage. This seems unfair.

[jsuereth]

Good point, and not the real intention here.

The agent-aided meant we may want to start leveraging agents within open-telemetry to define transitions at definition time, and publicizing these transitions for everyoen to consume without agents. We cannot have agent-in-the-loop on hot paths, that's intractable. Leveraging an agent to help improve the coverage of how many version transitions we can safely provide is what this was intended to mean.

The true goal is that we have an open model and easy-to-use transition "definition" that folks can consume to translate between versions of schema, or even (if we're successful) between different schema-urls entirely.

That's still experimental / long-term exploration for the project. For now, we have targeted version-bump transitions as something we believe we can provide automatic transition model that should NOT be costly unfair for the ecosystem to adopt.

[mikeldking]

I'm not familiar with the weaver MCP server. How does this work?

[jsuereth]

Today it's pretty light. You run weaver mcp and it will load in a registry and allow the agent to interact with a version of semantics and lookup the model, definitions, notes, etc. It can also run live-check to enforce conformance on the version. We hope to expand this further to be more of an aide when defining conventions and allow more ad-hoc agent-initiated flows, e.g. "Help me create a semantic convention registry from this integration test"