[AutoOps] Allow overriding parts of the agent collector configuration via spec.config/configRef

[pkoutsovasilis]

Summary

Adds spec.config and spec.configRef fields to AutoOpsAgentPolicy, giving users a supported way to tune the AutoOps agent's OpenTelemetry collector configuration.

Closes #9395

Background

The AutoOps agent's OTel/metricbeat config was rendered from a single hardcoded Go template into a controller-owned ConfigMap. The reconciler overwrote Data on every loop, so any manual edit was silently reverted. Users who needed to tune operational knobs - queue sizing, additional processors, custom exporters - had no supported path.

Every other ECK stack component already exposes a config / configRef pair (Beats, Agent, Logstash, EnterpriseSearch, EPR). This PR brings AutoOps in line with that pattern.

Design

The original config template is split into three:

• autoOpsBaselineConfig - user-tunable defaults (currently the sending_queue settings). Users can override any field here via spec.config or spec.configRef.
• autoOpsDefaultModulesConfig - the operator's standard autoops_es module definitions (Metrics + Templates). Injected only when neither spec.config nor spec.configRef defines at least one autoops_es module. When the user supplies their own autoops_es modules, this step is skipped entirely, giving them full control over which data is collected.
• autoOpsMandatoryConfig - operator-owned structural config (env-var references, OTLP exporter wiring, healthcheck extension). Applied last and always wins.

Merge order: baseline → userCfg → userSecretCfg → defaultModules → mandatory

This means users can freely tune the baseline and extend it (add custom modules, exporters, pipelines), while the operator's critical wiring is always re-asserted at the end - no validation layer needed. common.ParseConfigRef is always called unconditionally so the dynamic Secret watch lifecycle is managed correctly regardless of whether configRef is set.

What users can do

The merge uses ucfg.AppendValues semantics: scalar fields are overridden by the last write, and list fields accumulate entries. Because the mandatory config is merged last, users can extend but not replace operator-owned lists.

Supported today:

spec:
  config:
    exporters:
      otlphttp:
        sending_queue:
          queue_size: 104857600   # tune backpressure (issue #9501)
    exporters:
      debug:
        verbosity: detailed       # add a custom exporter
    service:
      pipelines:
        logs/debug:               # add an additional pipeline
          receivers: [metricbeatreceiver]
          exporters: [debug]
    receivers:
      metricbeatreceiver:
        metricbeat:
          modules:
            - module: autoops_es  # replace operator defaults with a custom autoops_es module
              period: 30s
              metricsets: [cluster_health]

When the user supplies at least one autoops_es module, the operator's built-in Metrics and Templates modules are not injected - the user's list is used as-is. This allows changing the period, metricsets, or dropping individual modules entirely. Non-autoops_es modules added by the user are appended alongside the operator's defaults (which are still injected in that case).

Or via a Secret reference:

spec:
  configRef:
    secretName: my-autoops-overrides   # Secret key: autoops_es.yml

The following fields are always injected by the operator and cannot be overridden: Elasticsearch connection details (hosts, SSL paths, token), OTLP exporter endpoint and Authorization header, and the healthcheck extension on port 13133.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/api-reference/main.md

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[pkoutsovasilis]

@pickypg — would appreciate your eyes on the baseline/mandatory config split in this PR.

The original monolithic template has been split into two:

• autoOpsBaselineConfig — currently only the sending_queue defaults. Users can override or extend anything here via spec.config / spec.configRef.
• autoOpsMandatoryConfigTemplate — rendered per-ES (needs the SSL/TLS context) and merged last so its values always win. Contains everything else: the two autoops_es metricbeat modules, processors (token injection), OTLP endpoint and Authorization env-var references, service pipeline wiring, and the full healthcheckv2 extension including the readiness probe port. The merge order is baseline → userCfg → userSecretCfg → mandatory, using standard ucfg.AppendValues semantics throughout.

The main question for you: does the mandatory set look right? Specifically:

• Are there fields currently in mandatory that should be user-overridable (e.g. service.telemetry.logs.encoding, module period/metricsets)?
• Are there fields currently in the baseline that should be locked down instead?

A consequence of this split is that users cannot change the period or metricsets of the operator's built-in modules — they can only append new module entries alongside them. Happy to revisit the split if that's considered a blocker.

[pickypg]

LGTM with one question.

[pickypg]

My only question is how well this merges things like:

receivers:
  metricbeatreceiver:
    metricbeat:
      modules:
        # Metrics
        - module: autoops_es
          hosts: ${env:AUTOOPS_ES_URL}
{{- template "ssl" . }}
          period: 10s
          metricsets:
            - cat_shards
            - cluster_health
            - cluster_settings
            - license
            - node_stats
            - tasks_management
        # Templates
        - module: autoops_es
          hosts: ${env:AUTOOPS_ES_URL}
{{- template "ssl" . }}
          period: 24h
          metricsets:
            - cat_template
            - component_template
            - index_template

If I drop one of those - module blocks, will it continue to exist? If I want to override one, can I? What happens if I set (drops the # Templates section:

receivers:
  metricbeatreceiver:
    metricbeat:
      modules:
        # Metrics
        - module: autoops_es
          hosts: ${env:AUTOOPS_ES_URL}
{{- template "ssl" . }}
          period: 10s
          metricsets:
            - cat_shards
            - cluster_health
            - cluster_settings
            - license
            - node_stats
            - tasks_management

I think it will do the right thing and receivers.metricbeatreceiver.metricbeat.modules will be replaced with the new array, but I want to confirm.

[pickypg]

I missed in your comment this is covered:

A consequence of this split is that users cannot change the period or metricsets of the operator's built-in modules — they can only append new module entries alongside them. Happy to revisit the split if that's considered a blocker.

I don't think it's a showstopper, but it would be nice if we can make it so this can be overridden somehow. Specifically, some users may not want to share their templates for any reason and similarly may not want to share their tasks. In that scenario, I would like to be able to support them. I wonder if there's some simple mechanism where we can have an override flag that says "the config here is complete and you only need to provide mandatory changes" to it, where that would not include the metricsets.

[pkoutsovasilis]

Hey @pickypg, thanks for taking a look here!

The main constraint I can think of is that allowing users to take full ownership of the config is risky - the healthcheckv2 extension on port 13133 is what the readiness probe hits, so if a user removes or remaps it the pod never becomes ready. Similarly, the OTLP endpoint/Authorization header and the metricbeatreceiver processor (token injection) are wiring the operator manages and that users shouldn't touch.

For the modules specifically though, I think there's a middle ground: if the user defines at least one autoops_es module in their spec.config or spec.configRef, the operator skips injecting its own default modules (Metrics + Templates) entirely, giving the user full control over which data is collected and at what schedule. If the user defines no autoops_es modules at all (or only adds non-autoops_es modules), the operator's defaults (Metrics + Templates) are injected as before.

Regardless of whether modules come from the user or the operator defaults, the operator always injects hosts, ssl.verification_mode, ssl.certificate_authorities, and (when mTLS is required) ssl.certificate/ssl.key into every autoops_es module post-merge. The user-supplied values for those fields are always overridden. This way users control what is collected but the operator always controls how the connection to ES is established.

Does this split sound reasonable to you?

[pkoutsovasilis]

@pickypg I did implement the approach described above in d945645, let me know your thoughts

[pickypg]

Thank you. This looks great (especially reviewing the tests).

[pickypg]

Are there fields currently in the baseline that should be locked down instead?

Since users without ECK can override all of these fields, I don't really consider anything off limits for ECK.

[prodsecmachine]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

implementation changed

[pebrc]

LGTM in general. I am a bit worried about the complexity of the deep config inspection and manipulation. I and the 🤖 suggested a somewhat simpler approach inline. I think the viability depends on how common it is that users want to add new modules without touching the autops_es defaults or whether removing one of the autoops_es default modules is the main use case.

[pebrc]

These assertions verify presence but not order. With AppendValues semantics the user's module (merged in an earlier layer) will always appear before the operator's defaults (merged later). A future change that reorders the merge constants would break this property without any test catching it. Consider pinning the order:

// user's module was merged before operator defaults, so it comes first
assert.Equal(t, "some_other_module", asMap(t, modules[0])["module"])
assert.Equal(t, "autoops_es", asMap(t, modules[1])["module"])

[pkoutsovasilis]

ok why does the order matter here?

[pebrc]

I was thinking along the lines of avoiding unnecessary restarts, and making sure this test fails if we do refactor this in a way that changes the order, so that the restart becomes a conscious choice. But maybe you are right and for AutoOps agents this is less of a concern.

[pebrc]

The code here is correct, but the PR description's Design section is now stale: it says the original template was "split into two" and shows merge order as baseline → userCfg → userSecretCfg → mandatory. The second commit introduced autoOpsDefaultModulesConfig as a third constant and updated the actual order. Worth updating the PR description to say "split into three" and reflect the defaultModules step, so future readers aren't confused.

[pkoutsovasilis]

PR description updated

[pebrc]

This predicate introduces a pattern that doesn't exist anywhere else in ECK: unpacking config into map[string]any, walking a list, and matching entries by field value. For comparison, the codebase has three established patterns for combining user and operator config:

1. Map-path layering (Beat, Agent): merge user config, then merge operator-owned settings at fixed map paths last (beat/common/config.go).
2. Presence checks at fixed paths (EnterpriseSearch): HasChildConfig to decide whether to inject a default (enterprisesearch/config.go:198).
3. Wholesale list ownership (Logstash pipelines): faced with the same "list-valued config that ucfg can't merge into" problem, Logstash chose if userProvidedCfg != nil { return userProvidedCfg.Render() } — either the operator owns the list or the user does (logstash/pipeline.go).

To be clear about the split: injectModuleConnectionSettings below is justified — SSL paths are per-ES while one spec.config fans out to N clusters, so the operator must fix up connection settings post-merge; wholesale ownership can't solve that. But this predicate could be simplified to the established pattern 2:

const modulesPath = "receivers.metricbeatreceiver.metricbeat.modules"
hasUserModules := userCfg.HasChildConfig(modulesPath) || userSecretCfg.HasChildConfig(modulesPath)

i.e. "if you touch the module list, you own the module list." That collapses this function to two calls of an existing helper, removes one of the two hand-rolled traversals over the same untyped structure (drift between them is a future bug class), and gives users a contract that's easy to state. The only behavioral difference — a user adding a non-autoops_es module also takes ownership of the defaults — is arguably more predictable, not less.

Whichever predicate wins, the module-ownership semantics need to be user-visible: the CRD field description currently only says connection details "are always injected", and the PR description still says changing period/metricsets of the built-in modules is "Not supported" — which d945645 made supported. The most surprising behavior in this PR currently exists only in code comments.

[pkoutsovasilis]

what you say here, including the simplification proposal, is valid only if we accept the approach - if the user touches the modules then the user owns the modules list. To that end, if a user is able to fabricate an AutoOpsAgentPolicy configuration without any autoops_es module in it, what the point of this CRD? IMO, in the case of logstash, agent, beats yes the users should own everything because these CRDs need to be tailored according their needs. However, AutoOpsAgentPolicy CRD, is here to fulfil the need to run an AutoOps agent. In this PR we just want allow the users to configure any knobs they need to configure but this still has to be an autoops-oriented config. All that said, if you still think that we should change that, happy to do so, just let me know

[pebrc]

Fair point, you convinced me. Let's maybe make the behaviour clear in the config attribute godoc (and with that in our public API docs). Maybe something like:

supplying your own autoops_es module replaces the operator's built-in modules

[pkoutsovasilis]

added in d80b11f

[pebrc]

LGTM (not tested!). Let's document the module merging behaviour in the CRD.

[pkoutsovasilis]

@pickypg I would appreciate another review from you, to make sure that we provide enough flexibility but we don't break anything autoops-related, before I merge this

[pkoutsovasilis]

buildkite test this -f p=gke,E2E_TAGS=autoops -m s=9.3.6,s=9.4.3-SNAPSHOT,s=9.5.0-SNAPSHOT

[pebrc]

I ran this branch through a manual test pass on a local kind cluster (operator image docker.elastic.co/eck-ci/eck-operator:pr-9507-4137dfd5, 1-node ES 9.2.0 with TLS, agent 9.2.1, dummy Cloud Connect secret — the CCM 401 is expected and doesn't affect config plumbing).

Test matrix

#	Scenario	Result
1	Baseline, no spec.config	✅ Both operator modules present, per-ES SSL paths injected, sending_queue defaults intact
2	Tune sending_queue.queue_size via spec.config	✅ User value wins, config hash changes, deployment rolls, new pod ready
3	Attempt to override otlphttp.endpoint, Authorization header, healthcheck port	✅ Operator re-asserts all mandatory fields; user values never reach the rendered config
4	User supplies own autoops_es module with bogus hosts/ssl/CA path	✅ Operator defaults dropped (single user module remains), period/metricsets preserved, bogus connection settings all replaced with operator-managed values
5	Switch to configRef, then mutate the Secret contents	✅ Dynamic watch fires; ConfigMap updated within seconds without touching the policy
6	Set both config and configRef	✅ Webhook rejects with both Forbidden errors
7	configRef Secret missing the autoops_es.yml key	✅ Warning event with clear message, policy phase Error — and the agent keeps running on the last-good config (nice failure containment)
8	Rendering determinism: 5+ forced reconciles	✅ ConfigMap Data byte-stable, config hash unchanged, zero pod restarts
9	Delete policy	✅ ConfigMap/Deployment GC'd via owner refs, no dangling-watch errors in operator logs

The module-ownership semantics and connection-settings injection both held up under adversarial input (scenario 4): user-supplied hosts: http://wrong:9200, ssl.verification_mode: none and a bogus CA path were all replaced with the operator-managed values in the rendered config.

Findings

1. Integer fidelity on the inline spec.config path. Setting queue_size: 104857600 inline renders as queue_size: 1.048576e+08 in the ConfigMap — JSON unmarshalling of spec.config forces numbers to float64, and the yaml round-trip in NewCanonicalConfigFrom doesn't restore integer form. The collector tolerates it (the pod rolled and came up ready), and the configRef path renders clean integers since Secret contents are parsed as YAML text. Cosmetic today, and likely shared by every ECK component using the same code path — so arguably pre-existing rather than a defect of this PR — but users diffing the rendered ConfigMap will be confused, and a stricter downstream parser could reject it.

2. Rendered output changes on upgrade → one-time fleet-wide agent restart. The old template emitted output.otelconsumer: {}; the ucfg render emits otelconsumer: null (plus key reordering). The agent parses it fine, but it means every existing AutoOps ConfigMap is rewritten once on operator upgrade, flipping the config hash and rolling every AutoOps agent pod. Expected behavior for a config-template change, but worth a line in the release notes/changelog.

3. Stale sample manifest. config/samples/autoops/autoops.yaml pins spec.version: 9.2.0, which the webhook now rejects ("lower than the lowest supported version of 9.2.1").

[pkoutsovasilis]

Rendered output changes on upgrade → one-time fleet-wide agent restart. The old template emitted output.otelconsumer: {}; the ucfg render emits otelconsumer: null (plus key reordering). The agent parses it fine, but it means every existing AutoOps ConfigMap is rewritten once on operator upgrade, flipping the config hash and rolling every AutoOps agent pod. Expected behavior for a config-template change, but worth a line in the release notes/changelog.

I don't like this transition from {} to null (maybe it is time to fix that issue in go-ucfg), I have to look more into that and if it disables something it shouldn't.

[pkoutsovasilis]

ok I found this

https://github.com/elastic/beats/blob/a829d56a0b9a475680780cce055a6c868a84de78/x-pack/libbeat/cmd/instance/beat.go#L96-L98

which means that this transition doesn't affect us at all, @pickypg is my understanding correct?

[pickypg]

LGTM. Thank you for making the metricsets overrideable.

[pickypg]

which means that this transition doesn't affect us at all, @pickypg is my understanding correct?

If this runs an agent successfully, then we should be fine. It's noteworthy that the configuration in question here is for the Elastic Agent running in OTel mode, and not Beats. In the Elastic Agent, in OTel mode, we do not specify the classic Beats configuration (even though a part of it bleeds in under mbreceiver). Instead, we specify an OTel configuration that, in this case, happens to use Metricbeat modules (hence the mbreceiver).

output.otelconsumer is not part of the configuration that you seem to be generating and I am confused by this AI comment as a result. For the configuration we're generating, I don't see where this comes from (maybe I missed it?), but if you take the "standard" config as an example, that does not appear anywhere in it:

https://github.com/elastic/elastic-agent/blob/main/internal/edot/samples/linux/autoops_es.yml

[pkoutsovasilis]

thanks for the review @pickypg , ok this config here https://github.com/elastic/cloud-on-k8s/blob/main/pkg/controller/autoops/configmap_test.go#L21-L97 is essentially what ECK has as the default configuration of an AutoOps agent (this one predates this PR). Compared to the configuration you posted above I can see they are not fully the same. So the question is, should we update the default configuration of ECK to match this one https://github.com/elastic/elastic-agent/blob/main/internal/edot/samples/linux/autoops_es.yml?