otelconf: provide strategy for supporting experimental features in schema

[codeboten]

Capturing this in a new issue to not lose sight of wanting to provide a strategy for how experimental features in the configuration schema are supported in otelconf moving forward as discussed in the 20-Nov SIG call.

Given this is an RC of v1 I want to ask what is the design/plan regarding backwards compatiblity and stability.

For instance OpenTelemetryConfiguration has InstrumentationDevelopment field. Should it be exported? This does not seem to be a stable field.

Do we want we eventually want to have Instrumentation instead of InstrumentationDevelopment?

That's a good question! The plan for the schema v1 is to include experimental support for:

• resource detection
• instrumentation
• otlp file exporter
• configurators for each signal
• prometheus exporter
• various samplers configuration (composite, probability)

The stability guarantee around these are near none (https://github.com/open-telemetry/opentelemetry-configuration/tree/main?tab=readme-ov-file#experimental-features). We could decide to not support these in the implementation at this time.

Alternatively we could put these in an experimental module and only unmarshal anything marked with the /development suffix if an option is set (something like WithExperimentalFeatures)

Thoughts?

Originally posted by @codeboten in #8026 (comment)

[codeboten]

I thought a bit more about what the strategy should look like here, providing individual options for things that would need to be set that are experimental would give us the flexibility to decide how much of the experimental functionality we want to provide an API for, but it would end up causing us to have to provide an API of some sort anyways so i'm not sure how much we would gain here.

[codeboten]

I opened a PR that shows how we could include a otelconf/x package that supports experimental features. The change worth looking at is probably this one: dfc7608 as the others are really just everything thats in the 1.0.0 rc pull request.

This approach ends up creating a lot of duplicated code, but it is much simpler as most of the duplicated code is generated from the schema itself and updated via sed, so not a lot of manual work there. Not sure how much of the marshaling/unmarshaling code we could reuse across packages. I'd like to get more input from the SIG before i go down that route.

Pinging @pellared and @MrAlias for opinions as we talked about this in various conversations

[MrAlias]

I like the idea of separating experimental things out of the stable module.

How will otelconf/x be used? Is this something where a user will setup their SDK with otelconf/x or otelconf depending on if they want experimental features or not, or is there a dependency between the two packages?

[codeboten]

How will otelconf/x be used? Is this something where a user will setup their SDK with otelconf/x or otelconf depending on if they want experimental features or not, or is there a dependency between the two packages?

That's a good question. From the point of view of managing less code, i would prefer if otelconf/x could include the model from otelconf and add whatever was experimental on top of it to avoid the duplication.

I could imagine as a user, the better UI would be to import otelconf and be able to pass in a WithExperimentalFeatures flag in ParseYAML which could return a model that could be either OpenTelemetryConfigurationStableOnly or OpenTelemetryConfigurationExperimental, not needing to import a separate package.

It's possible a otelconf/model package would give us more flexibility in the future.

[MrAlias]

I like the approach where a user would import otelconf/x explicitly to use the functionality there. That way, similar to the Go semantic versioning approach, they make an active choice to use a version of code that is unstable.

If the stable portions of the API are also in there, that means that when the feature they want to use becomes stable, all they need to do to upgrade is change the package path. Right?

[MrAlias]

If we bleed the otelconf/x package into otelconf by adding a dependency to support WithExperimentalFeatures it will also mean that our stable package will always depend on our unstable one. Likely also poisoning the stable package with instability.