Balancing Innovation and Stability in OpenTelemetry instrumentations

[StarpTech]

Description

Hi, we use OpenTelemetry and the go instrumentation heavily in production. They work great. Kudos to everyone who contributes.

I opened this ticket to start a discussion about how new features and breaking changes in semconv can be handled better. Let me elaborate with a simple use case. We use this library to instrument server applications. Our platform relies on the exact attributes the application sends to us to provide reports and analytics. We don't control what server version users will use. If attributes are changed, this will result in an outage of our service. Right now, I need to go through every minor release to ensure it. Here is a concrete example. This situation is similar when we talk about the introduction of new features, e.g., additional metrics or attributes.

Now, you could think that our use case is too advanced to rely on the base instrumentation. I agree, but also copying and modifying the code is not easy because code generation is used here. I could fork the library, but I think we should talk about a more flexible solution instead.

I propose to decouple semconv and what attributes are used inside the instrumentation so the user can decide what fields are really needed. This can be done by a hook system. A very logical example are Client IP attributes. The current OTLP http implementation tracks IP addresses by default and there is no way to opt-out. Many companies aren't allowed to track such sensitive information. Redacting them is not supported as a built-in feature by the exporter and even if it were supported, selecting the right semconv attributes is tricky and error-prone. Ideally, I don't start tracking them. (In case you're looking for a way to redact information anyway, I can recommend the https://github.com/MrAlias/redact library).

When it comes to new functionality, I'd wish we would enforce to use feature flags. Example: In the recent versions of OTLP http implementation metrics have been introduced. We can't use the latest release due to the reasons above and we can't benefit from other improvement because we can't opt out from metric collection.

We know that this library, according to SemVer, is below v1 and it might introduce breaking changes whenever it needs, but it makes adaptation for a production use case very hard. I also believe that this issue will persist once we have reached v1.