Argo CD not detecting or applying changes in structured spec.config.processors field for OpenTelemetry Collector CR

[Iamkishore486]

I’m observing that Argo CD is not detecting or applying changes made to nested fields inside the spec.config section of an OpenTelemetry Collector custom resource, even though the configuration is defined as structured YAML (not a multiline string).

• Updated a nested parameter inside the OpenTelemetry Collector CR:

spec:
  config:
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 77

• Argo CD behavior:

argocd app manifests <app-name> → shows updated value

argocd app diff <app-name> → no output

Application remains in Synced state

No sync is triggered

• However, in the cluster below command still shows the old value

kubectl get opentelemetrycollector servicemonitor -o yaml

• What I expected
  Argo CD should detect a difference in the nested field (spec.config.processors.memory_limiter.limit_percentage) and mark the application as OutOfSync, triggering a sync to apply the updated configuration.

• What actually happens

Argo CD does not detect any difference and does not attempt to apply the updated configuration, even though the rendered manifests clearly contain the new value.

• Confirmation

The spec.config field is defined as structured YAML, not a multiline string (|)

No ignoreDifferences or resource.customizations are configured

No errors are observed in Argo CD logs

• Environment

Argo CD version: v3.3.2

Kubernetes version: v1.34.3

Application type: Helm 

Resource: OpenTelemetry Collector CRD

• Additional details
  No ignoreDifferences or resource.customizations are configured.

• Workarounds observed

Forcing a sync (argocd app sync --force) applied changes 

Adding/changing annotations forces Argo CD to detect a diff

• Questions

Is Argo CD expected to detect changes in nested structured fields under spec.config for CRDs like OpenTelemetry Collector?

Could this behavior be related to how Argo CD performs diffing on CRDs with loosely defined schemas?

Are there any known limitations or recommended configurations for handling such cases?

[abhay1999]

Great question — this is a known pain point with CRDs that have loosely-typed or deeply nested fields like spec.config in the OpenTelemetry Collector.

Root Cause

The most likely cause is that the OpenTelemetry Operator’s admission webhook is normalizing spec.config before it is stored in etcd. When Argo CD runs its diff, it compares the desired manifest (from Git/Helm) against the live resource (from the cluster). If the webhook rewrites or reorders the nested fields during admission, the live state in etcd ends up structurally different from what Argo CD rendered — even though the semantics are the same.

The clue is your observation:

• argocd app manifests → shows the new value (desired state from Git/Helm)
• argocd app diff → no output (Argo CD thinks desired == live)
• kubectl get ... -o yaml → shows the old value

This means Argo CD’s diff engine is comparing the rendered manifest against a normalized/mutated live state and finding them equal, even though the actual cluster object is stale.

How to Confirm

Run:

argocd app diff <app-name> --local <path-to-manifests>

Also check if the OTel Operator has a mutating webhook:

kubectl get mutatingwebhookconfigurations | grep opentelemetry

Recommended Fix

Use resource.customizations in your Argo CD ConfigMap to tell the diff engine how to handle this CRD:

# argocd-cm ConfigMap
resource.customizations: |
  opentelemetry.io/OpenTelemetryCollector:
    ignoreDifferences: |
      jsonPointers:
      - /spec/config/processors

Or, if you want Argo CD to correctly detect real changes, configure managedFieldsManagers to exclude the operator’s manager from the diff:

resource.compareoptions: |
  ignoreManagedFieldsManagers:
  - opentelemetry-operator

This tells Argo CD to ignore fields last-written by the operator’s webhook, so only your Git-sourced changes are considered.

Answers to Your Questions

1. Yes, Argo CD should detect changes in nested structured fields — but only when the live resource in etcd actually reflects those changes. If a webhook is normalizing the field before storage, the diff will be empty.
2. This is likely CRD schema + webhook normalization, not a fundamental Argo CD limitation.
3. Configuring ignoreManagedFieldsManagers or resource.customizations is the recommended path. You may also want to file an issue with the OpenTelemetry Operator if the webhook is silently discarding valid config changes.

Hope this helps unblock you!

[Iamkishore486]

Thank you @abhay1999 for your suggestions and I appreciate your time and efforts in responding to the discussion. I tested both of your suggestions individually and combiningly but unfortunately both didn't work. I still see the same behaviour in argocd, Last sync still not picking latest commits.