AppController: error calculating server side diff with 2.10-rc1

[prune998]

Describe the bug

When enabling the server side diff in the Application, the app stays out-of-sync with an error:

Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff: serverSideDiff error: error converting json string to unstructured: unmarshal error: invalid character 'c' looking for beginning of value

The Application is synced fine when setting the option back to false:

kind: Application
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=false,IncludeMutationWebhook=true

To Reproduce

• install ArgoCD 2.10-rc1
• create an application enabling ServerSideDiff:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true
  name: devops-cicd-argocd-infrastructure-addons
  namespace: argocd
spec:
  destination:
    name: central-cluster
    namespace: argocd-infrastructure-addons
  ignoreDifferences:
  - group: apps
    jqPathExpressions:
    - .spec.template.spec.containers[].env[].valueFrom.resourceFieldRef.divisor
    - .spec.template.spec.initContainers[].env[].valueFrom.resourceFieldRef.divisor
    kind: Deployment
  project: infrastructure
  source:
    path: deployments/k8s/overlays/qa/central-cluster
    repoURL: https://gitlab/devops/cicd/argocd-infrastructure-addons.git
    targetRevision: deployment
  syncPolicy:
    automated:
      allowEmpty: true
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true

The repo contains a kustomized application that use the Helm Inflator:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
  - name: crossplane
    repo: https://charts.crossplane.io/stable
    releaseName: crossplane
    namespace: crossplane
    # version: 0.1.0 # use latest
    includeCRDs: false #no CRD in the chart, they are created by crossplane Operator
    valuesFile: ../../../base/crossplane/values.yaml
    valuesMerge: override
    valuesInline:
      args: 
        - --debug
      serviceAccount:
        customAnnotations: 
          iam.gke.io/gcp-service-account: crossplane@me.iam.gserviceaccount.com
    

resources:
  - ../../../base/configmaps #this creates a test configmap
  - ../../../base/crossplane # this creates the namespace and providers

Note:

• setting IncludeMutationWebhook=false but keeping ServerSideDiff=true shows the same behaviour so it's not linked to the Weebhook.

• commenting out the helm chart from the kustomization.yaml file resolved the issue. But the helm chart is working fine without the server-side diff option...

  ---
  apiVersion: kustomize.config.k8s.io/v1beta1
  kind: Kustomization
  
  resources:
    - ../../../base/configmaps #this creates a test configmap
    - ../../../base/crossplane # this creates the namespace and providers
  

• using the minecraft chart shows no problem.
  The Crossplane chart does generates sync errors because of the divisor added by K8s that is not part of the chart, so the ignoreDifferences is added in the App. The server-side sync issue is the same with or without this ignoreDifferences option. In fact, I was expecting that the server-side diff would solve this diff issue even without the ignoreDifferences.

Expected behavior

The App should deploy just fine, which it does with the same Application when setting the option ServerSideDiff=false

Version

v2.10.0-rc1+9ad5cd6.dirty (This is a custom build 2.10-rc1 from release-2.10 branch)

kubectl  version
Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.7-gke.1056000

kustomize version
v5.3.0

Logs

Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff: serverSideDiff error: error converting json string to unstructured: unmarshal error: invalid character 'c' looking for beginning of value

[leoluz]

@prune998 I can't access the repo in https://gitlab/devops/cicd/argocd-infrastructure-addons.git
Can you please provide a minimal Application configuration that reproduces the issue so I can inspect?

[prune998]

The application is on a private repo that I can't share, but you could simply create an application in your test setup and targert a repo that containes the provided kustomization.yaml file...

ex:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true
  name: server-side-diff-error
  namespace: argocd
spec:
  destination:
    name: in-cluster
    namespace: server-side-diff-error
  ignoreDifferences:
  - group: apps
    jqPathExpressions:
    - .spec.template.spec.containers[].env[].valueFrom.resourceFieldRef.divisor
    - .spec.template.spec.initContainers[].env[].valueFrom.resourceFieldRef.divisor
    kind: Deployment
  project: default
  source:
    path: deployments/k8s
    repoURL: <put your repo here>
    targetRevision: main
  syncPolicy:
    automated:
      allowEmpty: true
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true

Then the file deployments/k8s/kustomization.yaml

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
  - name: crossplane
    repo: https://charts.crossplane.io/stable
    releaseName: crossplane
    namespace: crossplane
    # version: 0.1.0 # use latest
    includeCRDs: false #no CRD in the chart, they are created by crossplane Operator
    valuesMerge: override
    valuesInline:
      args: 
        - --debug
      serviceAccount:
        customAnnotations: 
          iam.gke.io/gcp-service-account: crossplane@me.iam.gserviceaccount.com
    

resources:
  - ns.yaml

and the file deployments/k8s/ns.yaml:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    argocd.argoproj.io/sync-options: ServerSideApply=true
    argocd.argoproj.io/sync-wave: "0"
  labels:
    kubernetes.io/metadata.name: crossplane
  name: crossplane

[leoluz]

I suspect that you are facing a problem with namespace creation + server-side apply issue which is reported in #13874.

Can you please try disabling Server-Side apply but keep ServerSide Diff enabled?

[prune998]

testing, even if the namespace already exist and contains some resources...
Actually, the whole helm chart is applied... it's just the sync diff that seems to be failing...
The test should not take long (waiting for the app to be re-created from the appset)

[prune998]

I tested:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=false
  ...
    syncOptions:
    - CreateNamespace=true
    - ServerSideApply=false

And it failed with the same error.

Going to create a repo with the helm chart + everything needed...

[prune998]

testing with https://github.com/prune998/argocd-2.10-ssd now

[prune998]

I made 2 more push, and I effectively had to add "ServerSideApply=false" else nothing was deploying (even the namespace that I explicitelly create...)

Turning "ServerSideDiff=true" still show the same error

Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff: serverSideDiff error: error converting json string to unstructured: unmarshal error: invalid character 'c' looking for beginning of value

[prune998]

I tried updating the chart to solve the divisor issue and I also cleaned the whole namespace to start fresh... the same issue happens, and no resources are deployed.
I don't understand why the minecraft chart would work and not the crossplane one...

The modified helm chart is in the divisor branch

[prune998]

I tried removing things from the chart, and it seems the clusterRoles and ClusterRoleBindings are the issue. Everything else in the chart sync well.
Note that Argo DOES create the CR and CRB without problem... but fail to sync anyways

[leoluz]

@prune998 I am able to reproduce the bug. I'll work on the fix and reply back.
Thanks for the investigation!

[anandf]

The fix done as part of PR argoproj/gitops-engine#563 seems to fix the error. The application sync was not complete, but still the error `namespace "crossplane" is not found" was resolved.

Used the application provided as part of the reproducer and additionaly enabled helm in argocd-cm config map using the following command.

 kubectl patch cm argocd-cm -n argocd -p '{"data":{"kustomize.buildOptions": "--enable-helm"}}'

argocd/server-side-diff-error  in-cluster  crossplane  default  Unknown  Healthy  Auto-Prune  ComparisonError  https://github.com/anandf/argocd-example-apps.git  helm-inflator  add-helm-inflator-example