[2.10 RC] Namespace not created in ArgoCD v2.10.0-rc1 when using CreateNamespace syncOption

[hlastras]

Description

We encountered an issue with ArgoCD v2.10.0-rc1 where the Application resource doesn't create the required namespace when syncPolicy.syncOptions includes CreateNamespace=true and ServerSideApply=true. It works if I remove option ServerSideApply=true

Steps to Reproduce:

• Set up a local Kubernetes cluster using Kind (I've tested with version 1.27.x and 1.29.0).

• Install ArgoCD v2.10.0-rc1 (following instructions from the release notes):

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.10.0-rc1/manifests/install.yaml

• Apply the following Application manifest example using kubectl apply -f:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: helloworld
  namespace: argocd
spec:
  destination:
    namespace: helloworld
    server: https://kubernetes.default.svc
  project: default
  source:
    path: services/dev-x/helloworld
    repoURL: https://github.com/hlastras/testargo.git
    targetRevision: HEAD
  syncPolicy:
    automated:
      allowEmpty: true
      prune: true
    retry:
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 1m
      limit: 2
    syncOptions:
    - ApplyOutOfSyncOnly=true
    - Validate=true
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground
    - ServerSideApply=true
    - FailOnSharedResource=true

And it fails with the error namespaces "helloworld" not found

This are part of the logs of the ApplicationController. On it, I can see it tries to create the Namespace, but any error.

time="2024-01-11T12:30:16Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/helloworld
time="2024-01-11T12:30:16Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: helloworld)" application=argocd/helloworld
time="2024-01-11T12:30:16Z" level=debug msg="Generating Manifest for source {https://github.com/hlastras/testargo.git services/dev-x/helloworld HEAD nil nil nil nil  } revision HEAD"
time="2024-01-11T12:30:16Z" level=info msg="GetRepoObjs stats" application=argocd/helloworld build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=715 unmarshal_ms=714 version_ms=0
time="2024-01-11T12:30:16Z" level=debug msg="Retrieved live manifests" application=argocd/helloworld
time="2024-01-11T12:30:16Z" level=debug msg=app.status.expired application=argocd/helloworld useDiffCache=false
time="2024-01-11T12:30:16Z" level=info msg="Initiated automated sync to '23f1c8c7aea1b9e054430152f105f90e1806038d'" application=helloworld dest-namespace=helloworld dest-server="https://kubernetes.default.svc" reason=OperationStarted spec="{&ApplicationSource{RepoURL:https://github.com/hlastras/testargo.git,Path:services/dev-x/helloworld,TargetRevision:HEAD,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,} {https://kubernetes.default.svc helloworld  false} default &SyncPolicy{Automated:&SyncPolicyAutomated{Prune:true,SelfHeal:false,AllowEmpty:true,},SyncOptions:[ApplyOutOfSyncOnly=true Validate=true CreateNamespace=true PrunePropagationPolicy=foreground ServerSideApply=true FailOnSharedResource=true],Retry:&RetryStrategy{Limit:2,Backoff:&Backoff{Duration:5s,Factor:*2,MaxDuration:1m,},},ManagedNamespaceMetadata:nil,} [] [] <nil> []}" type=Normal
time="2024-01-11T12:30:16Z" level=info msg="Start Update application operation state"
time="2024-01-11T12:30:16Z" level=info msg="Initiated automated sync to '23f1c8c7aea1b9e054430152f105f90e1806038d'" application=argocd/helloworld
time="2024-01-11T12:30:16Z" level=info msg="Updated sync status:  -> OutOfSync" application=helloworld dest-namespace=helloworld dest-server="https://kubernetes.default.svc" reason=ResourceUpdated spec="{&ApplicationSource{RepoURL:https://github.com/hlastras/testargo.git,Path:services/dev-x/helloworld,TargetRevision:HEAD,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,} {https://kubernetes.default.svc helloworld  false} default &SyncPolicy{Automated:&SyncPolicyAutomated{Prune:true,SelfHeal:false,AllowEmpty:true,},SyncOptions:[ApplyOutOfSyncOnly=true Validate=true CreateNamespace=true PrunePropagationPolicy=foreground ServerSideApply=true FailOnSharedResource=true],Retry:&RetryStrategy{Limit:2,Backoff:&Backoff{Duration:5s,Factor:*2,MaxDuration:1m,},},ManagedNamespaceMetadata:nil,} [] [] <nil> []}" type=Normal
time="2024-01-11T12:30:16Z" level=info msg="Updated health status:  -> Missing" application=helloworld dest-namespace=helloworld dest-server="https://kubernetes.default.svc" reason=ResourceUpdated spec="{&ApplicationSource{RepoURL:https://github.com/hlastras/testargo.git,Path:services/dev-x/helloworld,TargetRevision:HEAD,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,} {https://kubernetes.default.svc helloworld  false} default &SyncPolicy{Automated:&SyncPolicyAutomated{Prune:true,SelfHeal:false,AllowEmpty:true,},SyncOptions:[ApplyOutOfSyncOnly=true Validate=true CreateNamespace=true PrunePropagationPolicy=foreground ServerSideApply=true FailOnSharedResource=true],Retry:&RetryStrategy{Limit:2,Backoff:&Backoff{Duration:5s,Factor:*2,MaxDuration:1m,},},ManagedNamespaceMetadata:nil,} [] [] <nil> []}" type=Normal
time="2024-01-11T12:30:16Z" level=info msg="Completed Update application operation state"
time="2024-01-11T12:30:16Z" level=info msg="updated 'argocd/helloworld' operation (phase: Running)" appNamespace=argocd application=helloworld project=default
time="2024-01-11T12:30:16Z" level=info msg="Initialized new operation: {&SyncOperation{Revision:23f1c8c7aea1b9e054430152f105f90e1806038d,Prune:true,DryRun:false,SyncStrategy:nil,Resources:[]SyncOperationResource{},Source:nil,Manifests:[],SyncOptions:[ApplyOutOfSyncOnly=true Validate=true CreateNamespace=true PrunePropagationPolicy=foreground ServerSideApply=true FailOnSharedResource=true],Sources:[]ApplicationSource{},Revisions:[],} { true} [] {2 &Backoff{Duration:5s,Factor:*2,MaxDuration:1m,}}}" application=argocd/helloworld
time="2024-01-11T12:30:16Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: helloworld)" application=argocd/helloworld
time="2024-01-11T12:30:16Z" level=debug msg="Generating Manifest for source {https://github.com/hlastras/testargo.git services/dev-x/helloworld HEAD nil nil nil nil  } revision 23f1c8c7aea1b9e054430152f105f90e1806038d"
time="2024-01-11T12:30:16Z" level=info msg="GetRepoObjs stats" application=argocd/helloworld build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=5 unmarshal_ms=5 version_ms=0
time="2024-01-11T12:30:16Z" level=debug msg="Retrieved live manifests" application=argocd/helloworld
time="2024-01-11T12:30:16Z" level=debug msg=app.status.expired application=argocd/helloworld useDiffCache=false
time="2024-01-11T12:30:16Z" level=info msg=Syncing application=argocd/helloworld skipHooks=false started=false syncId=00004-Yjdpx
time="2024-01-11T12:30:16Z" level=info msg="Tasks from managed resources" application=argocd/helloworld resourceTasks="[Sync/0 resource apps/Deployment:helloworld/helloworld nil->obj (,,)]" syncId=00004-Yjdpx
time="2024-01-11T12:30:16Z" level=info msg="tasks from hooks" application=argocd/helloworld hookTasks="[]" syncId=00004-Yjdpx
time="2024-01-11T12:30:16Z" level=info msg="Update successful" application=argocd/helloworld
time="2024-01-11T12:30:16Z" level=info msg="Reconciliation completed" application=argocd/helloworld dedup_ms=0 dest-name= dest-namespace=helloworld dest-server="https://kubernetes.default.svc" diff_ms=3 fields.level=3 git_ms=715 health_ms=0 live_ms=0 patch_ms=22 setop_ms=8 settings_ms=0 sync_ms=0 time_ms=764
time="2024-01-11T12:30:16Z" level=info msg="Tasks (dry-run)" application=argocd/helloworld syncId=00004-Yjdpx tasks="[PreSync/0 resource /Namespace:/helloworld nil->obj (,,), Sync/0 resource apps/Deployment:helloworld/helloworld nil->obj (,,)]"
time="2024-01-11T12:30:16Z" level=info msg="Running tasks" application=argocd/helloworld dryRun=true numTasks=2 syncId=00004-Yjdpx
time="2024-01-11T12:30:16Z" level=info msg=Applying application=argocd/helloworld dryRun=true syncId=00004-Yjdpx task="PreSync/0 resource /Namespace:/helloworld nil->obj (,,)"
time="2024-01-11T12:30:16Z" level=info msg="Applying resource Namespace/helloworld in cluster: https://10.96.0.1:443, namespace: "
time="2024-01-11T12:30:16Z" level=info msg="{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"creationTimestamp\":null,\"name\":\"helloworld\"},\"spec\":{},\"status\":{}}"
time="2024-01-11T12:30:16Z" level=info msg=Applying application=argocd/helloworld dryRun=true syncId=00004-Yjdpx task="Sync/0 resource apps/Deployment:helloworld/helloworld nil->obj (,,)"
time="2024-01-11T12:30:16Z" level=info msg="Applying resource Deployment/helloworld in cluster: https://10.96.0.1:443, namespace: helloworld"
time="2024-01-11T12:30:16Z" level=info msg="{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"helloworld\"},\"name\":\"helloworld\",\"namespace\":\"helloworld\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"helloworld\"}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"helloworld\"}},\"spec\":{\"containers\":[{\"args\":[\"echo Hello \\u0026\\u0026 sleep 600\"],\"command\":[\"/bin/sh\",\"-c\"],\"image\":\"busybox\",\"name\":\"busybox\"}]}}}}"
time="2024-01-11T12:30:16Z" level=info msg="Applying resource Deployment/helloworld in cluster: https://10.96.0.1:443, namespace: helloworld"
time="2024-01-11T12:30:16Z" level=info msg="{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"labels\":{\"app.kubernetes.io/instance\":\"helloworld\"},\"name\":\"helloworld\",\"namespace\":\"helloworld\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"helloworld\"}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"helloworld\"}},\"spec\":{\"containers\":[{\"args\":[\"echo Hello \\u0026\\u0026 sleep 600\"],\"command\":[\"/bin/sh\",\"-c\"],\"image\":\"busybox\",\"name\":\"busybox\"}]}}}}"
time="2024-01-11T12:30:17Z" level=info msg="Apply failed" application=argocd/helloworld dryRun=true message="namespaces \"helloworld\" not found" syncId=00004-Yjdpx task="Sync/0 resource apps/Deployment:helloworld/helloworld nil->obj (,,)"
time="2024-01-11T12:30:17Z" level=info msg="Adding resource result, status: 'SyncFailed', phase: 'Failed', message: 'namespaces \"helloworld\" not found'" application=argocd/helloworld kind=Deployment name=helloworld namespace=helloworld phase=Sync syncId=00004-Yjdpx
time="2024-01-11T12:30:17Z" level=info msg="Updating operation state. phase: Running -> Failed, message: '' -> 'one or more objects failed to apply (dry run)'" application=argocd/helloworld syncId=00004-Yjdpx
time="2024-01-11T12:30:17Z" level=info msg="sync/terminate complete" application=argocd/helloworld duration=232.690292ms syncId=00004-Yjdpx
time="2024-01-11T12:30:17Z" level=info msg="Start Update application operation state"
time="2024-01-11T12:30:17Z" level=info msg="Completed Update application operation state"
time="2024-01-11T12:30:17Z" level=info msg="updated 'argocd/helloworld' operation (phase: Running)" appNamespace=argocd application=helloworld project=default
time="2024-01-11T12:30:17Z" level=info msg="Skipping retrying in-progress operation. Attempting again at: 2024-01-11T12:30:27Z" application=argocd/helloworld
time="2024-01-11T12:30:17Z" level=info msg="Skipping retrying in-progress operation. Attempting again at: 2024-01-11T12:30:27Z" application=argocd/helloworld

Observations:

• The logs from the ApplicationController do not show any errors related to namespace creation.
• Reverting to ArgoCD v2.9.3 with the same setup and manifest works as expected.
• Removing ServerSideApply=true from syncPolicy.syncOptions in v2.10.0-rc1 resolves the issue.

Expected Behavior:

The Application resource should create the specified namespace when CreateNamespace=true is set, as it does in version 2.9.3.

[rbreeze]

Related (possibly duplicate): #13874

[34fathombelow]

Thank you @hlastras for reporting this and the detailed description. I can also confirm this bug, we discussed this during the Argo Contributors meeting. We will get some eyes on this.

[hlastras]

Related (possibly duplicate): #13874

I've tried searching for similar issues but missed the one you mentioned. My apologies.

In our tests, the issue seems related but not identical. In our Application, we use syncOptions like CreateNamespace and ServerSideApply. In the stable version 2.9.3, this works, but in version 2.10.0-rc1, it doesn't.

Interestingly, if I add the annotation argocd.argoproj.io/sync-options: ServerSideApply=true to the Deployment and only use CreateNamespace in the Application, it fails in both versions 2.9.3 and 2.10.0-rc1. This suggests a new problem introduced in 2.10.0-rc1, adding to an existing issue.

Additionally, the issue you mentioned discusses a workaround by adding managedNamespaceMetadata: {} to the Application. This seems to work up until version 2.9.3, but breaks in 2.10.0-rc1 too.

To summarize:

v2.9.3:

• Application with syncOptions: CreateNamespace=true and Deployment with argocd.argoproj.io/sync-options: ServerSideApply=true -> fails
  • Adding managedNamespaceMetadata: {} in Application -> works
• Application with syncOptions: CreateNamespace=true and ServerSideApply=true -> works

v2.10.0-rc1:

• Application with syncOptions: CreateNamespace=true and Deployment with argocd.argoproj.io/sync-options: ServerSideApply=true -> fails
  • Adding managedNamespaceMetadata: {} in Application -> fails
• Application with syncOptions: CreateNamespace=true and ServerSideApply=true -> fails

I hope I've conducted the tests correctly. If not, apologies in advance 🙏

[anandf]

The issue seems to have caused after this specifc gitops-engine version upgrade.
https://github.com/argoproj/argo-cd/pull/16232/files

These 2 PRs in GitOps Engine project are the suspects
argoproj/gitops-engine#548
argoproj/gitops-engine#546

[anandf]

reverting the PR argoproj/gitops-engine#546 seems to resolve the issue.
argoproj/gitops-engine@master...anandf:gitops-engine:fix_namespace_create_error

Need to look deeper in the changes that is causing the error

[anandf]

Root Cause:

When server side apply is set to true, dry run is run in server mode, if the dry run fails in server mode, then it is re-run in client mode. But the server side apply is still set. This is not a supported configuration in kubectl when --dry-run is set to client, --server-side flag is not supported.

# kubectl apply -k kustomize-guestbook --dry-run=client --server-side -n guestbook -o json
error: --dry-run=client doesn't work with --server-side (did you mean --dry-run=server instead?)

Solution

When kubectl apply --dry-run=server --server-side fails, then re-run the dry run in client mode and with server-side flag removed as below

# kubectl apply -k kustomize-guestbook --dry-run=client -n guestbook -o json

Created this PR argoproj/gitops-engine#563 for fixing this

[leoluz]

@hlastras This issue should be now addressed by argoproj/gitops-engine#564
Are you able to give it a try with the latest master build to confirm?

[hlastras]

Hi @leoluz
I have tried to deploy master (latest) without changes in our setup and now it works without problems. 👏

Thank you very much for your quick work