Should file configuration merge environment variable configuration?

[jack-berg]

The conversation about whether file configuration should completely ignore the sdk environment variable scheme came up in #3744, but that PR doesn't actually contain any language related to this.

The original file configuration OTEP stated:

Interpret the configuration model and return SDK TracerProvider, MeterProvider, LoggerProvider which strictly reflect the configuration object's details and ignores the opentelemetry environment variable configuration scheme.

As mentioned here, file configuration doesn't actually contain language describing this behavior. It was included originally included in #3437 but was lost in the PR review shuffle - accidentally, not in response to feedback.

@tedsuo argues in favor file configuration respecting env vars with:

The common expectation among developers is that env vars will automatically overwrite config parameters. If we do it the other way, I am concerned that until the end of time we will have a steady stream of users lodging issues about this and then becoming very frustrated when we explain that they need to modify the config file to use an env var.

The reason that these users will be upset is that they are in a situation where having to define the env vars in the config file is a non-starter. Their use case is that for some reason they either can't get at or are not allowed to modify the config file template, but they really need to override a parameter.

For reference, this is an issue that has come up on many OSS projects I have been involved with where it is common to have both operators and application developers wanting to configure the same thing. Often it's some kind of emergency situation where the person with the rights to change the config file is unavailable.

I should note that of course the opposite situation could be true, where for some reason you want to disable an env var but don't have access to it. But it's probably easier to give users the ability to disable an env var via the config file than it is to give them the ability to disable a config parameter via an env var.

@MrAlias argues in favor of ignoring env vars with:

To be fair, from experience, you're going to get people complaining either way. If you choose environment variable priority over a configuration users will complain that their deployment was altered and failed when an environment variable was set that took precedence.

However, if you make environment variables take precedence you will also need to make some pretty sever and subjective choices on how they are mapped to a config. Do the BSP environment variables apply to all batch span processors or just one? Is the sampler environment variable use for all tracer providers in the config, even ones that specify alternate samplers? Should propagators be merged or overridden? If an exporter is defined by environment, does that stop the console logging exporter used for debugging as well as all the other exporters defined in configuration?

Ultimately, I think the current changes are going to be the most appropriate. They allow users to make their own choice in precedence without making subjective choices for them on how to map things. If a user want environment variables to take precedence, all they need to do is use the OTel environment keys in the related parts of their configuration. In doing so they will answer each of the above questions their own way.

@trask supports the feeling of users expecting env vars to override file configuration, but also says merging configuration from multiple sources is hard:

This is my feeling as well, especially for things like:

OTEL_SDK_DISABLED
OTEL_RESOURCE_ATTRIBUTES
OTEL_SERVICE_NAME
OTEL_LOG_LEVEL
OTEL_PROPAGATORS
OTEL_TRACES_SAMPLER_ARG
I totally understand the nightmare that is merging configuration from multiple sources though.

I wonder if we would have created many of the other env vars (e.g. OTEL_BSP_, OTEL_BLRP_) if we had configuration file support from the beginning? And if so, maybe we can deprecate those other env vars in favor of configuration file?

This topic came up several times during the lengthly review of the file configuration OTEP. Below are links to a number of and relevant points:

open-telemetry/oteps#225 (comment)

Layering of config as described below would make it more difficult to reason about what my program is actually being run with.

Additionally, perhaps give them a helper config file that uses env var substitution in the right places so that they can migrate easily (and still get a warning until they get rid of env variables and move everything to the config file).

open-telemetry/oteps#225 (comment)

What about 'Solution 3: fail when both environment configuration and file configuration are present'?

I think we could log a warning when we detect this, but failing is too strong. Consider the implications if a user is operating in an environment they don't fully control (i.e. where an ops team configures environment variables by default which they extend / layer on top of).

open-telemetry/oteps#225 (comment)

I have a (rather strong) opinion that setting set via env vars has higher priority (takes precedence) over a setting set via configiuration file and it should be marked as a goal.

Any scheme where environment variables have priority over a config file will require some sort of standard mapping between the environment variable schema and file config scheme. IMO, its impossible to define such a mapping which is intuitive in all cases, so better not to try.

Nothing is forcing users to use file based config - its opt in. If they do opt in, they're opting into the documented behavior in which the config file represents the source of truth for configuration. If they wish to customize the experience with additional layers / overrides, they have a couple of tools:

1. They can use the fact the Configure(config) API accepts a config model as an argument, and provide their own customizations to the model after the initial parsing of the file via Parse(file). An example of such a customization would be to interpret environment variables and apply them to the model in a way they decide makes sense.
2. They can use environment variable substitution to reference environment variables directly in a configuration file.

---

Update 3/15/2024

The current state of this issue is:

• The configuration working group proposed and executed a process to move forward after several months of failing to reach consensus
• As a part of that process, the configuration maintainers reviewed proposals and have recommended moving forward with option 2.b as described in this comment
• As a part of that process, people who disagree with the recommendation can / should escalate to a TC decision

---

Update 3/28/2024

Please see this comment updating the status of the issue:

Per @tedsuo’s request, we discussed this issue in the 3/24/27 TC meeting and have made a decision:
Generally, we will follow @trask's comment, proceeding with this PR with a few changes:

• Rename OTEL_CONFIG_FILE to OTEL_EXPERIMENTAL_CONFIG_FILE, reflecting the fact that the semantics around how the value of env var are subject to breaking changes as the file configuration spec and schema continue to evolve.
• Ensure that env vars which don’t interop with file config are deprecated when file config is ready for stabilization, reflecting that we do not want to recommend multiple competing configuration stories. This could be ensured via an explicit note in the markdown, or a blocking issue - both achieve the same effect. Deprecate environment variables which do not interoperate with file config when ready for stabilization #3967
• Ensure that file config has an interop where platforms (i.e. Azure functions, otel operator, etc) contribute to config. We should proceed with #3948 without being prescriptive about how that mechanism works. In the TC meeting, 4 distinct solutions were discussed which had different tradeoffs and limitations. It is clear that we still need to learn more about the requirements and constraints of this use case and let the findings inform the solution. The config working group should prioritize this discussion, but an answer shouldn’t block this PR. We should open a new issue to track the requirements and discuss solutions, and ensure that we treat that issue as blocking for any sort of stabilization effort (although it should ideally be solved much sooner). Ensure that file config has solution for platforms which contribute to config #3966

[jack-berg]

My point of view is that while I understand that there is precedence in other systems to have environment variables override other configuration sources, we shouldn't do it. The environment variable configuration scheme does not map cleanly to file configuration, leading to unintuitive behavior when trying to merge. A partial mapping of some of the environment variables is also unintuitive. If we give support environment variable substitution and default values (#3744) that will allow us to provide a file configuration "template" to users as a starting point, complete with comments and env var substitution references to the environment variable scheme with default fallbacks. Note this was the conclusion of OTEP conversation open-telemetry/oteps#225 (comment).

What's not to like? Users start out with the template that essentially is performing the merging of the environment variable scheme with the file configuration scheme (where it makes sense). The template has comments which reinforce that if they delete the env var substitution references, those environment variables will not be considered during parse / create. This should make migrating from environment variable schema to file configuration smooth, while also allowing the implementation to have simple intuitive rules that are easy to explain and which are logically consistent. While some users will inevitably ignore the documentation and expect environment variables to trump file configuration anyway, they'll surely be able to understand the motivation when pointed to the docs.

[yurishkuro]

I agree with @jack-berg (#3752 (comment)). It's better to have a clean break in both user interface and SDK components interface: components should no longer support env vars directly, they should always take parameters from a config object, and the config object can support env vars as placeholders, but without any semantic meaning attached to their names (it's up to the user to pick those names). It is minimalistic design with clean encapsulation and separation of responsibilities. In that sense env vars do override config file, but only because user explicitly writes them this way, not because of some magic mapping between env var name and the exact position in the config.

Having said that, before we go this way we need to have a very clear migration strategy to avoid introducing breaking changes. The default config template that pulls in all the env vars already defined in the spec is a good approach. What's not clear to me there is whether it requires support for conditionals, because, for instance, in order to have a place in the config to refer to some var related to OTLP exporter, we need to know that OTLP exporter is what the user actually wants to use. It may be possible to accommodate using the declare/use separation used in the Collector config, i.e. the template config will always have a section for OTLP exporter (and all its env vars), but the tracer config portion may not reference the exporter as "I want to use it".

So basically I think we need a working prototype of the config to validate that this approach is workable.

Another migration concern is whether it can be incremental - there are many components in the SDK and requiring they all to be upgraded to config object style of initialization (vs env vars) before anything can be released is going to be a problem. The above approach with a default config template seems like it could be incremental.

One more open question - how such default config template will play with the ability to minimize runtime dependencies of the SDK?

[brunobat]

Please beware that popular frameworks have their config systems in place for years, creating yet another config file might be nice from the OTel point of view but not for the frameworks that already instaciate an OTel SDK of their own.

Placing all possible configs in a OTel config will require frameworks to scan for services there, as example, if we want them to work on native mode.
Frameworks already do many other things like, providing defaults, validations and value transformations.
All this is already done and would be bypassed by the new OTel Config file.

From my point of view, the properties supplier from the SDK must have the highest priority, higher then the OTel config file.
More, there shouldn't be any configuration of the SDK that cannot be performed by using the properties supplier.

In relation to the env. vars., in all frameworks I can remember, they take precedence over any other kind of configuration. OTel shouldn't be special.

[jack-berg]

From my point of view, the properties supplier from the SDK must have the highest priority, higher then the OTel config file.
More, there shouldn't be any configuration of the SDK that cannot be performed by using the properties supplier.

These are java specific concepts @brunobat. Can you generalize this feedback to the spec level?

[jack-berg]

Please beware that popular frameworks have their config systems in place for years, creating yet another config file might be nice from the OTel point of view but not for the frameworks that already instaciate an OTel SDK of their own.
Frameworks already do many other things like, providing defaults, validations and value transformations.
All this is already done and would be bypassed by the new OTel Config file.

Popular frameworks have their own config systems. That's great - if those frameworks want to make opentelemetry a first class citizen, they can evolve those frameworks to be able to able to configure the things opentelemetry users expect (the flat scheme provided by environment variables very quickly runs into problems expressing very real config scenarios). But not all users buy into these frameworks - do we leave these users behind, only providing a suboptimal environment variable schema?

File configuration isn't and doesn't need to be the only configuration option. It doesn't erase the environment variable scheme, or programatic configuration which enables all sorts of alternative configuration mechanisms like those provided by frameworks. But we do need a language agnostic way to fully express the desired configuration of an SDK.

Is the request to do something different with file configuration or to not have a file configuration option at all?

Placing all possible configs in a OTel config will require frameworks to scan for services there, as example, if we want them to work on native mode.

This appears to be no different than the problem we face today in opentelemetry-java with the SPIs to load custom exporters. With opentelemetry-sdk-extension-autoconfigure, you can set OTEL_TRACES_EXPORTER=foo, and if there is a ConfigurableSpanExporterProvider implementation on the classpath corresponding to foo, autoconfigure will use it to create the corresponding SpanExporter.

In relation to the env. vars., in all frameworks I can remember, they take precedence over any other kind of configuration. OTel shouldn't be special.

Below I've listed several examples where trying to merge environment variables with file configuration yields unintuitive / unexpected results. For me to take this request seriously, I need to see proposals on how these situations would be resolved:

Simple Priority

We have the same options in env vars and config file, but different values. Which wins? Easy enough - environment variables always win. Although, we'll no doubt have users complaining that what they see in their config file isn't reflected in reality.

Env vars

OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

File config

tracer_provider:
  processors:
    - batch:
         exporter:
           otlp: 
             endpoint: http://some-other-endpoint:4317

Conflict example 1

We have env variable information which can be merged with file configuration such that both are true. In this case, env variables specify to use the OTLP exporter, while file configuration says zipkin. One person might expect to see the OTLP exporter overwrite zipkin. Another might expect to see spans exported to both OTLP and zipkin. Yet another might expect to see only zipkin.

Env vars

OTEL_TRACES_EXPORTER=otlp

File config

tracer_provider:
  processors:
    - batch:
         exporter:
           zipkin: http://localhost:9411/api/v2/spans

Conflict example 2

In this case we have configuration which can be merged, but doing so will almost certainly yield the wrong result. The environment variables specify the OTLP trace endpoint, assuming the http/protobuf default protocol, which includes the path. The config file specifies grpc protocol. If the environment variable takes priority over file configuration, http/protobuf variant of the endpoint specified via environment variable will be used to configure the OTLP grpc exporter specified in file config, causing an error.

Env vars

OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://some-other-endpoint:4318/foo/bar/v1/traces

File config

tracer_provider:
  processors:
    - batch:
         otlp:
             endpoint: http://some-endpoint:4317
             protocol: grpc

Conflict example 3

In this case we see how the flat nature of the environment variable scheme falls short when trying to merge with the structure of a config file. The config file specifies that the sdk should export spans to two different endpoints, but an environment variable is specified which sets the OTLP endpoint to something else. If you override both from the config file you're almost certainly not doing what the user would want. And if you change the effective config to result in a single OTLP exporter with the endpoint in the environment variable, you're also almost surely not doing what the user wants. Presumably, they want to override one of the endpoints in the config file, but its impossible to know that they really want this, or which in the config file to override.

Env vars

OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=http://yet-another-endpoint:4318

File config

tracer_provider:
  processors:
    - batch:
         otlp:
             endpoint: http://some-endpoint:4318
    - batch:
         otlp:
             endpoint: http://some-other-endpoint:4318

I can keep listing examples, but the point is that I don't know of any strategy for overlaying environment variables on top of a configuration file that will make sense to most users (yet alone all) in most cases. Instead, we should aim to give simple primitives which can be combined to accommodate simple and complex use cases:

• Provide an in memory representation of a config model
• Provide an operation to parse a config file to an in memory config model
• Provide an operation to create a configured SDK from an in memory config model
• Provide a way to perform environment variable substitution in a config file
• Provide an environment variable with a value pointing to a config file, which triggers file configuration

Most uses will will be fine just saying OTEL_CONFIG_FILE=/path/to/sdk-config.yaml, and be content with the simply ability to do environment variable substitution in a config file.

Users with complex requirements can combine these primitives in all sorts of interesting way:

• produce an in memory config model from some alternative source
• define merge logic for combining multiple in memory config models
• define logic for overlaying the environment variable scheme on top of a in memory config model in a way that makes sense for them

[brunobat]

These are java specific concepts @brunobat. Can you generalize this feedback to the spec level?

Yes, will send a PR to the sdk-configuration file.

The spec actually doesn't define any priority over config methods, just states "The SDK MUST provide a programmatic interface for all configuration". I agree with the statement.
This can be interpreted as the file config is just another SDK configuration method and shouldn't take precedence over the base generic programatic interface, which in the Java case is represented by the properties supplier.

The OTel file config must use the base generic programatic interface and shouldn't prevent the use of other existing or future configuration systems of the SDK... Which seems totally reasonable.

[brunobat]

Simple Priority

We have the same options in env vars and config file, but different values. Which wins? Easy enough - environment variables always win. Although, we'll no doubt have users complaining that what they see in their config file isn't reflected in reality.

Sure someone will complain, but overriding configs defined in files with env. vars. is widely accepted and common place in the container world.
It can be decided either way if properly documented, however not giving higher priority to env. vars. nowadays seems counter intuitive and agains the industry practice.

Conflict example 1

In this case the exporter should be OTLP and the endpoint the one specified in the file. Note that the endpoint shouldn't be specific to zipkin, but owned by the exporter.

These overrides happen all the time in a microservice. Files define the base config and env. vars. define the exceptions to particular attributes. I acknowledge that there can be many overrides.

Conflict example 2

Yes, the resulting merged config doesn't make sense, but the error can also happen with the file based config. Validation will be always needed and should be implemented to save people's time.

Conflict example 3

This is an excellent example of things that cannot be properly configured by env. vars. with the current abstraction, which leads me to the main point....

I think we are mixing the definition of a "programmatic interface for all configuration" with the definition of the file based configuration. This makes the programatic interface effectively useless for other configuration systems because the file config takes precedence and will include things that cannot be done in any other way.
The programmatic interface should be independent if we want to properly configure things with files, env. vars. and other config systems, as discussed above.

We should have a generic reusable programmatic interface for the config, a builder pattern for it, and only then the file format configuration for it. Merging and validating the config attributes should also be a task for the builder.

[jack-berg]

I think we are mixing the definition of a "programmatic interface for all configuration" with the definition of the file based configuration. This makes the programatic interface effectively useless for other configuration systems because the file config takes precedence and will include things that cannot be done in any other way.

Why does file configuration make the programmatic interface effectively useless?

File configuration is an abstraction that is built on top of the programmatic configuration interface. It will be natural to package file configuration in a separate artifact than the core SDK components it configures. By definition, it can not be more expressive than the programmatic interface since ultimately all options need to be translated to programmatic equivalents. The proposal in #3805 in which file configuration takes priority over the environment variable scheme is only true if the user opts into it by: 1. Including the necessary file configuration artifact. 2. Specifying OTEL_CONFIG_FILE=...

A framework that has its own configuration system can avoid file configuration entirely by not including the artifact, and / or by providing equivalent functionality. I.e. the framework provides its own format and schema for specifying configuration, and provides logic which parses, validates that configuration and uses the programmatic configuration interface to produce an SDK according to the configuration.

We should have a generic reusable programmatic interface for the config, a builder pattern for it, and only then the file format configuration for it.

💯 That's exactly what's happening.