Decide on Effective Config structure for language agents

[tigrannajaryan]

We need to decide how to record config in effectiveConfig field.

Copying notes that @breedx-splk wrote:

Formats may be different, depending on how the agent was initially configured
  - env vars vs declarative config
  - Jason thinks we agreed to use different formats, but I think Rasmus (and maybe others?)
    have pushed back and prefer a single format in all cases.
  - Jason was thinking `key=value\n` when env and filtered yaml when declarative config.

[RassK]

I think agents should report both anyway.

1. In case environment variables are used:

• report only .env, remote configuration option must be disabled

2. in case yaml (declarative configuration) is used:

• report .env when yaml contains envs
• report .yaml, this is a basis for remote configuration.

Notes: you can either use envs or yaml (+ envs included in the yaml).
https://opentelemetry.io/docs/languages/sdk-configuration/declarative-configuration/#environment-variables

All environment variables are ignored unless you explicitly add them to the config file.

3. Additionally we can report other file types as well for informational purpose, but those cannot be used for anything else. Eg .NET is using app/web.config with envs syntax.

[breedx-splk]

@RassK Let's limit the scope of this discussion ONLY to the reporting of effective config. We shouldn't be discussing remote configuration in this issue, and that's a separate concern. That said, I do disagree with some of what you've assumed above, but we shouldn't get into that here (please feel free to open a new issue to discuss).

[breedx-splk]

2. in case yaml (declarative configuration) is used:

• report .env when yaml contains envs
• report .yaml, this is a basis for remote configuration.

Notes: you can either use envs or yaml (+ envs included in the yaml). https://opentelemetry.io/docs/languages/sdk-configuration/declarative-configuration/#environment-variables

I do not think we should be sending two separate config "file" payloads in this case. What we SHOULD be sending instead is a limited/filtered representation of the yaml structure, with any environment vars expanded and any defaults or programmatic overrides also applied. The final yaml representation that we put on the wire should reflect what's effectively being used in the agent.

[tigrannajaryan]

The final yaml representation that we put on the wire should reflect what's effectively being used in the agent.

This is key.

As someone who is running the Collector I really want to see that final, effective config after all the transformations, env variable substitutions, etc are done. I want to see what Collector's components see, not what Collector's config reading code sees, because I am most interested in how components behave and the component behavior depends on that final config they are fed when the component is created.

[RassK]

Let's limit the scope of this discussion ONLY to the reporting of effective config.

I do not see those as completely separate topics. One is effectively basing on the other (at least on my mind). For sure you can use different paths to achieve something similar. Eg converting UIs to partial configs and sending those back instead of full set.

with any environment vars expanded

That's losing information you cannot reverse any more in the later stages. Secondly large transformation could not be OSS compatible way (at least not in the near terms).

I really want to see that final, effective config after all the transformations, env variable substitutions, etc are done.

If you operate with source of truth (and no transformations), you should be able to do all the transformations for the UI / UX needed in any time. The backing does not necessarily matter as far as you have all the information. I cannot imagine presenting any config anyway in the UI, this is not friendly UX for most (if it's going to differ from the original config, you make users to learn 2 different presentations of the same thing)

So far operating with yaml means you should be knowing all defaults, presets etc (thanks to schemas and versions). Since you should not bring in information from any other source than yaml or envs defined in yaml.

CC: @breedx-splk @tigrannajaryan

[tigrannajaryan]

If you operate with source of truth (and no transformations), you should be able to do all the transformations for the UI / UX needed in any time.

How are you suggesting this is done in practice? Do you suggest that the client sends the templated config, the default values, the env variable values that should be used in substitution and the backend is expected to perform the exact same transformation as the Collector does to compute the effective config? This is not going to work. The most likely result is that implementations of how config is computed by Collector and by the backend diverge and you will see incorrect effective config in the UI. Imagine that Collector behavior is different between versions and releases? Are we requiring the backend to precisely reproduce Collector behavior for every version?

If you mean some other approach then please describe.

[RassK]

I don't know much about collector's behaviour, I'll also have to double check if there are any complex evaluations in the dotnet agent's side. Or maybe you can give some real examples that are more complex than checking if the block is enabled and which sub options are overriden.

Currently can't imagine that the initial config is so off from what was declared. This is also the usual cause of security issues when defaults try to be too clever (causing unexpected things to be enabled).

The exceptional complex case I thought is if spec describes one thing and agent doesn't follow it. For different versions schema translations are already there - agent must report the correct schema version.

Additionally I guess it does not hurt if you report evaluated config and initial config both (following the same schema).

[aunshc]

@RassK can we begin with sending the evaluated config? Keeping the end user in mind, the outcome we are looking to drive is that they should be able to see the live resolved configuration to validate in cases they applied a certain configuration or troubleshoot if needed. I'm not sure I follow the need for sending the initial config as well for the backend to evaluate.

@tigrannajaryan do you agree?

[tigrannajaryan]

@aunshc I agree.

I don't mind the proposal @RassK has and I agree that there is value in showing the initial/unevaluated config. However, since we don't have clarity on how exactly to do it I suggest we don't get blocked on it, start with evaluated config and if there is a detailed proposal on how to add initial config, how to reflect default values, etc and show that in addition to evaluated config, that would be great and will provide a complete feature. We can add that later. Showing evaluated config now doesn't prevent that future.

[aunshc]

@RassK @seemk with a goal to start in a direction and iteratively make progress, can we decide to move forward with the following format for evaluated config in that case: key=value for env variables and yaml for declarative configuration.

[robsunday]

So it seems that we have initial agreement on the format of reported effective config, but there are some other items to be discussed.
Let’s focus on env vars config first. Just to summarize - we agreed so far on:

• Effective config file contains key/value pairs in format: key=value, each in a separate line.
• File should contain properties after all programmatic customizations are applied.

For now I see two items to discuss:

1. Which env vars should we report:
   a) predefined set of env vars. Maybe all env vars listed in Splunk docs for example? Maybe just a subset of it? (with default value if env var is not defined in the runtime environment)
   b) only the env vars defined in the runtime environment or added via customizations in the code.
   c) other option?
2. Content type used for effective config file sent to the server:
   a) “text/plain”
   b) “text/plain+properties” ?
   c) other option?

1a is the most complete solution, but it will also be the most expensive to implement and may need maintenance if env var is added/removed or default is changed.
1b seems to be easy to implement (at least in Java) and requires no maintenance, but on the other hand it does not provide complete picture of the configuration used.

My preference would be 1a and 2b. What do you think?

[Kielek]

2. related to format, I think that it should mark that it is splunk-developed format to be easily distinguish with potential common standard on the OTel side.

[robsunday]

So what you suggest is "text/plain+properties+splunk" ?

[robsunday]

Actually, I believe this format is an intermediate step, and the end goal is to use a format from the upstream, but I agree that it may be useful for some time - until upstream format gets defined and stabilized.