[confmap] - new feature flag for append merging strategy

[VihasMakwana]

Koanf's default merging strategy currently overrides static values such as slices, numbers, and strings. However, lists of components should be treated as a special case. This pull request introduces a new command line option to allow for merging lists instead of discarding the existing ones.

With this new merging strategy:

• All the lists are merged rather than replaced.
• The merging logic is name-aware, meaning that if components with the same name appear in both lists, they will only appear once in the final merged list.

Link to tracking issue

Related issues:

• Append components flag #8754
• Allow instantiating extensions in separate configs #8394
• [ConfMap] - Providing custom merge method for koanf's Map.Merge #10370

Testing

• Added

Documentation

• Added under readme.

Example

Consider the following configs,

# main.yaml
receivers:
  otlp/in:
processors:
  batch:
exporters:
  otlp/out:
extensions:
  file_storage:

service:
  pipelines:
    traces:
      receivers: [ otlp/in ]
      processors: [ batch ]
      exporters: [ otlp/out ]
  extensions: [ file_storage ]

# extra_extension.yaml
extensions:
  healthcheckv2:

service:
  extensions: [ healthcheckv2 ]

If you run the collector with following command,

otelcol --config=main.yaml --config=extra_extension.yaml --feature-gates=-confmap.enableMergeAppendOption

then the final configuration after config resolution will look like following:

# main.yaml
receivers:
  otlp/in:
processors:
  batch:
exporters:
  otlp/out:
extensions:
  file_storage:
  healthcheckv2:

service:
  pipelines:
    traces:
      receivers: [ otlp/in ]
      processors: [ batch ]
      exporters: [ otlp/out ]
  extensions: [ file_storage, healthcheckv2 ]

For backward compatibly, the default behaviour is not to merge lists. Users who want to explicitly merge lists can enable the command line option.

Note: I’d appreciate your feedback on this 🙏

[VihasMakwana]

@Aneurysm9 @evan-bradley @mx-psi can you take another look?

i've removed feature flag and added command line argument to control the paths user wants merged.

I'll work on the documentation once we've established a common ground.

[codecov]

Codecov Report

Attention: Patch coverage is 98.18182% with 1 line in your changes missing coverage. Please review.

Project coverage is 92.04%. Comparing base (72878b7) to head (7e731ce).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
confmap/resolver.go	85.71%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #12097      +/-   ##
==========================================
+ Coverage   92.02%   92.04%   +0.01%     
==========================================
  Files         469      470       +1     
  Lines       25318    25372      +54     
==========================================
+ Hits        23299    23353      +54     
  Misses       1612     1612              
  Partials      407      407              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mx-psi]

Thanks for taking this Vihas! I have two comments:

1. I think this is the kind of thing that is hard to get right on the first try, so I would want to do this under some sort of experimental flag.
2. I would like to do this if possible with little to no modification of stable confmap APIs. confmap is 1.0, it is a very important piece of the Collector, and if we get it wrong it would be quite difficult to reverse course.

[VihasMakwana]

@mx-psi Thanks for your comments!

1. Makes sesne. I will move it under experimental flag.
2. I understand. I will again take a fresh look and make sure we tick all the boxes.

[dmitryax]

LGTM

[dmitryax]

Please update the nolint comment to fix CI

[VihasMakwana]

A few of test cases are failing but they seem unrelated to my PR.

[VihasMakwana]

@mx-psi I have fixed the the go.mod and I'm not reverting extensionauth. can you please re-review?