How to get argocd to ignore CR schema validation against older CRD during diff calculation

[chiragjn]

K8s: 1.33
ArgoCD: 2.14.10

I am trying to work with the nvidia gpu-operator chart. We use ArgoCD's helm feature to apply the chart

The chart itself installs a CRD (ClusterPolicy) and a CR for it
First installation goes all good

Then when upgrading, if new fields have been added to CR, the sync fails at diff stage

Failed to compare desired state to live state: failed to calculate diff: error calculating structured merge diff: error building typed value from config resource: errors: .spec.dcgmExporter.service: field not declared in schema .spec.driver.kernelModuleType: field not declared in schema

If I understand this correctly, the new CR in the chart is being validated against the old CRD in the cluster.
I don't care about this validation because the chart itself has a pre-upgrade hook to upgrade the CRD but this diff calculation blocks the PreSync phase from starting

https://github.com/NVIDIA/gpu-operator/blob/cf2e878ce13e43119840d7f48cb54659eb2395ad/deployments/gpu-operator/templates/upgrade_crd.yaml#L56

---

Here is what my inital spec looks like

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: gpu-operator
  namespace: gpu-operator
spec:
  destination:
    namespace: gpu-operator
    server: https://kubernetes.default.svc
  project: default
  revisionHistoryLimit: 1
  source:
    chart: gpu-operator
    helm:
      releaseName: gpu-operator
    repoURL: https://helm.ngc.nvidia.com/nvidia
    targetRevision: v24.9.2
  syncPolicy:
    automated:
      prune: true
      selfHeal: false
    retry:
      limit: 0
    syncOptions:
      - RespectIgnoreDifferences=true
      - FailOnSharedResource=true
      - ApplyOutOfSyncOnly=true
      - ServerSideApply=true

And when trying to upgrade

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: gpu-operator
  namespace: gpu-operator
spec:
  destination:
    namespace: gpu-operator
    server: https://kubernetes.default.svc
  project: default
  revisionHistoryLimit: 1
  source:
    chart: gpu-operator
    helm:
      releaseName: gpu-operator
    repoURL: https://helm.ngc.nvidia.com/nvidia
    targetRevision: v25.3.1
  syncPolicy:
    automated:
      prune: true
      selfHeal: false
    retry:
      limit: 0
    syncOptions:
      - RespectIgnoreDifferences=true
      - FailOnSharedResource=true
      - ApplyOutOfSyncOnly=true
      - ServerSideApply=true

One possible solution might be to use helm's --disable-openapi-validation

#5366
#16144

But I believe the diff calculation happens before helm install stage is reached so it might not solve the problem

[chiragjn]

Sorry for the direct tag @alexmt @crenshaw-dev
Would really appreciate if some maintainer has seen such an issue before and can write an answer