Argo CD not picking up any applications

[elysweyr]

Hi,

I am currently trying to debug an issue on my cluster-scope Argo CD instance. After a sync it seems that all applications are vanished from the UI (not deleted from k8s itself but not detected by Argo CD anymore).

I've exec'ed into the argocd-server pod and logged in as admin.

argocd@cluster-argocd-server-5b64c8449d-dxvrs:~$ argocd app list
NAME  CLUSTER  NAMESPACE  PROJECT  STATUS  HEALTH  SYNCPOLICY  CONDITIONS  REPO  PATH  TARGET

The CLI commands shows the same behavior even though apps are already present and its namespaces are managed by Argo CD.

argocd@cluster-argocd-server-5b64c8449d-dxvrs:~$ argocd cluster get in-cluster
annotations:
  argocd.argoproj.io/refresh: "2025-06-07T12:03:04Z"
config:
  tlsClientConfig:
    insecure: false
connectionState:
  attemptedAt: "2025-06-07T12:13:08Z"
  message: Cluster has no applications and is not being monitored.
  status: Unknown
info:
  applicationsCount: 0
  cacheInfo: {}
  connectionState:
    attemptedAt: "2025-06-07T12:13:08Z"
    message: Cluster has no applications and is not being monitored.
    status: Unknown
labels:
  app.kubernetes.io/managed-by: cluster-argocd
  app.kubernetes.io/name: cluster-argocd-default-cluster-config
  app.kubernetes.io/part-of: argocd
name: in-cluster
namespaces:
- cluster-argocd
- cluster-bootstrap 
- ingress-nginx
- sealed-secrets
refreshRequestedAt: "2025-06-07T12:03:04Z"
server: https://kubernetes.default.svc

This confirms that the following namespaces are watched for updates:

namespaces:
- cluster-argocd
- cluster-bootstrap 
- ingress-nginx
- sealed-secrets

Let's list the Argo CD applications of cluster-bootstrap.

❯ kubectl get app -n cluster-bootstrap -o wide
NAME                        SYNC STATUS   HEALTH STATUS   REVISION                                   PROJECT
bootstrap-prod-mh-mgmt-01   Synced        Healthy         be7dc18a48e16afab798414917eb81bcc632fae0   default                        
self-bootstrapping                                                                                   default

❯ kubectl describe app -n cluster-bootstrap bootstrap-prod-mh-mgmt-01
Name:         bootstrap-prod-mh-mgmt-01
Namespace:    cluster-bootstrap
Labels:       <none>
Annotations:  <none>
API Version:  argoproj.io/v1alpha1
Kind:         Application
Metadata:
  Creation Timestamp:             2025-06-02T23:58:45Z                                                                                 
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2025-06-07T11:38:55Z                                                                                 
  Finalizers:
    resources-finalizer.argocd.argoproj.io
  Generation:  2047
  Owner References:
    API Version:           argoproj.io/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  ApplicationSet
    Name:                  bootstrap-clusters
    UID:                   6ea74100-6ed9-44b4-b861-45ed5aa57cd9
  Resource Version:        4480609
  UID:                     a7412649-5795-406f-8727-7429f41fb166
Spec:
  Destination:
    Name:       in-cluster
    Namespace:  argocd
  Project:      default
  Source:
    Helm:
      Value Files:
        ../../clusters/prod-mh-mgmt-01/values.yaml
    Path:             charts/1_template-cluster-appsets
    Repo URL:         [email protected]:elsysweyr/elys-mgmt-gitops.git                                                                      
    Target Revision:  HEAD
  Sync Policy:
    Automated:
      Prune:      true
      Self Heal:  true
    Sync Options:
      CreateNamespace=true
Status:
  Controller Namespace:  cluster-argocd
  Health:
    Status:  Healthy                                                                                                                   
  History:
    Deploy Started At:  2025-06-04T21:54:42Z
    Deployed At:        2025-06-04T21:54:45Z
    Id:                 0
    Initiated By:
      Automated:  true
    Revision:     11c6f5e33331e35398aeed641b69c0a0bc517a2e
    Source:
      Helm:
        Value Files:                                                                                                                           
          ../../clusters/prod-mh-mgmt-01/values.yaml                                                                                       
        Path:             charts/1_template-cluster-appsets                                                                                  
        Repo URL:         [email protected]:elsysweyr/elys-mgmt-gitops.git                                                                     
        Target Revision:  HEAD
    Deploy Started At:  2025-06-05T21:23:08Z                                                                                             
    Deployed At:        2025-06-05T21:23:08Z
    Id:                 1
    Initiated By:
      Automated:  true
    Revision:     03f0ac190b051c5a2e2aad57ae2306799a1027dd
    Source:
      Helm:
        Value Files:                                                                                                                           ../../clusters/prod-mh-mgmt-01/values.yaml
      Path:             charts/1_template-cluster-appsets                                                                                  
      Repo URL:         [email protected]:elsysweyr/elys-mgmt-gitops.git                                                                      
      Target Revision:  HEAD
  Operation State:
    Finished At:  2025-06-05T21:23:08Z
    Message:      successfully synced (all tasks run)
    Operation:
      Initiated By:
        Automated:  true
      Retry:
        Limit:  5
      Sync:
        Prune:     true
        Revision:  03f0ac190b051c5a2e2aad57ae2306799a1027dd
        Sync Options:
          CreateNamespace=true
    Phase:       Succeeded
    Started At:  2025-06-05T21:23:08Z
    Sync Result:
      Resources:
        Group:       argoproj.io
        Hook Phase:  Running
        Kind:        Application
        Message:     application.argoproj.io/provision-prod-mh-mgmt-01 created                                                               
        Name:        provision-prod-mh-mgmt-01
        Namespace:   cluster-bootstrap
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1alpha1
      Revision:      03f0ac190b051c5a2e2aad57ae2306799a1027dd
      Source:
        Helm:
          Value Files:
            ../../clusters/prod-mh-mgmt-01/values.yaml
        Path:             charts/1_template-cluster-appsets                                                                                  
        Repo URL:         [email protected]:elsysweyr/elys-mgmt-gitops.git                                                                      
        Target Revision:  HEAD
  Reconciled At:          2025-06-06T23:48:56Z
  Resources:
    Group:      argoproj.io
    Kind:       Application
    Name:       provision-prod-mh-mgmt-01
    Namespace:  cluster-bootstrap
    Status:     Synced
    Version:    v1alpha1                                                                                                               
  Source Type:  Helm                                                                                                                   
  Summary:                                                                                                                             
  Sync:                                                                                                                                  
    Compared To:
      Destination:
        Name:       in-cluster
        Namespace:  argocd
      Source:
        Helm:
            Value Files:
                ../../clusters/prod-mh-mgmt-01/values.yaml
        Path:             charts/1_template-cluster-appsets
        Repo URL:         [email protected]:elsysweyr/elys-mgmt-gitops.git
        Target Revision:  HEAD 
      Revision:             be7dc18a48e16afab798414917eb81bcc632fae0
      Status:               Synced
Events:                   <none>

Let's confirm the configured permissions:

❯ kubectl get clusterrolebinding -n cluster-bootstrap | grep argocd
argocd-cluster-resources-manager-binding                           ClusterRole/argocd-cluster-resources-manager                                  4d15h
argocd-namespace-reader-binding                                    ClusterRole/argocd-namespace-reader                                           8d
argocd-operator-controller-manager-service-system:auth-delegator   ClusterRole/system:auth-delegator                                             8d
argocd-operator.v0.13.0--3bbzPNnVjGgsaVTwYFfe2fFcHREZA62UeZfuvf    ClusterRole/argocd-operator.v0.13.0--3bbzPNnVjGgsaVTwYFfe2fFcHREZA62UeZfuvf   8d
argocd-operator.v0.13.0-mZsJ6QOLlWZmEmsaPyUYsMfHqqK2rLRx69gNJ      ClusterRole/argocd-operator.v0.13.0-mZsJ6QOLlWZmEmsaPyUYsMfHqqK2rLRx69gNJ     8d
                                                                                                                                     ~
❯ kubectl get rolebinding -n cluster-bootstrap
NAME                                           ROLE                                                AGE
argocd-namespace-manager-binding               ClusterRole/argocd-namespace-manager                8d
cluster-argocd-argocd-application-controller   Role/cluster-argocd-argocd-application-controller   4d14h
cluster-argocd-argocd-server                   Role/cluster-argocd-argocd-server                   4d14h
cluster-argocd_cluster-bootstrap               Role/cluster-argocd_cluster-bootstrap               4d14h

❯ kubectl describe clusterrole argocd-namespace-manager
Name:         argocd-namespace-manager
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources                               Non-Resource URLs  Resource Names  Verbs
  ---------                               -----------------  --------------  -----
  configmaps                              []                 []              [*]
  events                                  []                 []              [*]
  persistentvolumeclaims                  []                 []              [*]
  pods                                    []                 []              [*]
  secrets                                 []                 []              [*]
  serviceaccounts                         []                 []              [*]
  services                                []                 []              [*]
  daemonsets.apps                         []                 []              [*]
  deployments.apps                        []                 []              [*]
  replicasets.apps                        []                 []              [*]
  statefulsets.apps                       []                 []              [*]
  applications.argoproj.io                []                 []              [*]
  applicationsets.argoproj.io/status      []                 []              [*]
  applicationsets.argoproj.io             []                 []              [*]
  appprojects.argoproj.io                 []                 []              [*]
  argocds.argoproj.io                     []                 []              [*]
  cronjobs.batch                          []                 []              [*]
  jobs.batch                              []                 []              [*]
  ingresses.networking.k8s.io             []                 []              [*]
  rolebindings.rbac.authorization.k8s.io  []                 []              [*]
  roles.rbac.authorization.k8s.io         []                 []              [*]
   
❯ kubectl describe clusterrole argocd-namespace-reader
Name:         argocd-namespace-reader
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources                                                       Non-Resource URLs  Resource Names  Verbs
  ---------                                                       -----------------  --------------  -----
  namespaces                                                      []                 []              [create update patch get list watch]
  nodes                                                           []                 []              [get list watch]
  persistentvolumes                                               []                 []              [get list watch]
  mutatingwebhookconfigurations.admissionregistration.k8s.io      []                 []              [get list watch]
  validatingadmissionpolicies.admissionregistration.k8s.io        []                 []              [get list watch]
  validatingadmissionpolicybindings.admissionregistration.k8s.io  []                 []              [get list watch]
  validatingwebhookconfigurations.admissionregistration.k8s.io    []                 []              [get list watch]
  customresourcedefinitions.apiextensions.k8s.io                  []                 []              [get list watch]
  apiservices.apiregistration.k8s.io                              []                 []              [get list watch]
  applications.argoproj.io                                        []                 []              [get list watch]
  applicationsets.argoproj.io                                     []                 []              [get list watch]
  appprojects.argoproj.io                                         []                 []              [get list watch]
  certificatesigningrequests.certificates.k8s.io                  []                 []              [get list watch]
  flowschemas.flowcontrol.apiserver.k8s.io                        []                 []              [get list watch]
  prioritylevelconfigurations.flowcontrol.apiserver.k8s.io        []                 []              [get list watch]
  gatewayclasses.gateway.networking.k8s.io                        []                 []              [get list watch]
  accesscontrolpolicies.hub.traefik.io                            []                 []              [get list watch]
  etcdsnapshotfiles.k3s.cattle.io                                 []                 []              [get list watch]
  ingressclasses.networking.k8s.io                                []                 []              [get list watch]
  ipaddresses.networking.k8s.io                                   []                 []              [get list watch]
  servicecidrs.networking.k8s.io                                  []                 []              [get list watch]
  runtimeclasses.node.k8s.io                                      []                 []              [get list watch]
  olmconfigs.operators.coreos.com                                 []                 []              [get list watch]
  operators.operators.coreos.com                                  []                 []              [get list watch]
  clusterrolebindings.rbac.authorization.k8s.io                   []                 []              [get list watch]
  clusterroles.rbac.authorization.k8s.io                          []                 []              [get list watch]
  priorityclasses.scheduling.k8s.io                               []                 []              [get list watch]
  csidrivers.storage.k8s.io                                       []                 []              [get list watch]
  csinodes.storage.k8s.io                                         []                 []              [get list watch]
  storageclasses.storage.k8s.io                                   []                 []              [get list watch]
  volumeattachments.storage.k8s.io                                []                 []              [get list watch]
  secrets                                                         []                 []              [list watch]

❯ kubectl describe clusterrole argocd-cluster-resources-manager
Name:         argocd-cluster-resources-manager
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources                                                     Non-Resource URLs  Resource Names  Verbs
  ---------                                                     -----------------  --------------  -----
  mutatingwebhookconfigurations.admissionregistration.k8s.io    []                 []              [*]
  validatingwebhookconfigurations.admissionregistration.k8s.io  []                 []              [*]
  customresourcedefinitions.apiextensions.k8s.io                []                 []              [*]
  ingressclasses.networking.k8s.io                              []                 []              [*]
  clusterrolebindings.rbac.authorization.k8s.io                 []                 []              [*]
  clusterroles.rbac.authorization.k8s.io                        []                 []              [*]
     

Argo CD's logs aren't much of a help either ...

stern . -n cluster-argocd | grep -v "received unary call" | grep -v "finished unary call"
[...]
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T11:50:46Z" level=info msg="Notifying 1 settings subscribers: [0xc00055e9a0]"
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T11:51:07Z" level=info msg="kustomize version --short" dir= execID=88649
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T11:51:07Z" level=info msg=Trace args="[kustomize version --short]" dir= operation_name="exec kustomize" time_ms=12.814191000000001
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T11:51:07Z" level=info msg="helm version --client --short" dir= execID=45468
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T11:51:07Z" level=info msg="received streaming call /application.ApplicationService/Watch" grpc.method=Watch grpc.request.claims="{\"exp\":1749383467,\"iat\":1749297067,\"iss\":\"argocd\",\"jti\":\"71a1bb9e-c657-4808-9c64-d5de6572f973\",\"nbf\":1749297067,\"sub\":\"admin\"}" grpc.request.content="resourceVersion:\"4490545\" selector:\"\" appNamespace:\"\" " grpc.service=application.ApplicationService grpc.start_time="2025-06-07T11:51:07Z" span.kind=server system=grpc
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T11:51:07Z" level=info msg=Trace args="[helm version --client --short]" dir= operation_name="exec helm" time_ms=121.962503
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T11:51:09Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2025-06-07T11:51:07Z" grpc.time_ms=1969.222 span.kind=server system=grpc
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T11:53:47Z" level=info msg="received streaming call /application.ApplicationService/Watch" grpc.method=Watch grpc.request.claims="{\"exp\":1749383467,\"iat\":1749297067,\"iss\":\"argocd\",\"jti\":\"71a1bb9e-c657-4808-9c64-d5de6572f973\",\"nbf\":1749297067,\"sub\":\"admin\"}" grpc.request.content="resourceVersion:\"4494069\" selector:\"\" appNamespace:\"\" " grpc.service=application.ApplicationService grpc.start_time="2025-06-07T11:53:47Z" span.kind=server system=grpc
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T11:54:02Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2025-06-07T11:53:47Z" grpc.time_ms=15068.194 span.kind=server system=grpc
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T11:54:03Z" level=info msg="received streaming call /application.ApplicationService/Watch" grpc.method=Watch grpc.request.claims="{\"exp\":1749383467,\"iat\":1749297067,\"iss\":\"argocd\",\"jti\":\"71a1bb9e-c657-4808-9c64-d5de6572f973\",\"nbf\":1749297067,\"sub\":\"admin\"}" grpc.request.content="resourceVersion:\"4494069\" selector:\"\" appNamespace:\"\" " grpc.service=application.ApplicationService grpc.start_time="2025-06-07T11:54:03Z" span.kind=server system=grpc
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T11:54:08Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2025-06-07T11:54:03Z" grpc.time_ms=5156.827 span.kind=server system=grpc
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T11:54:42Z" level=info msg="received streaming call /application.ApplicationService/Watch" grpc.method=Watch grpc.request.claims="{\"exp\":1749383467,\"iat\":1749297067,\"iss\":\"argocd\",\"jti\":\"71a1bb9e-c657-4808-9c64-d5de6572f973\",\"nbf\":1749297067,\"sub\":\"admin\"}" grpc.request.content="resourceVersion:\"4495250\" selector:\"\" appNamespace:\"\" " grpc.service=application.ApplicationService grpc.start_time="2025-06-07T11:54:42Z" span.kind=server system=grpc
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T11:54:42Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2025-06-07T11:54:42Z" grpc.time_ms=391.962 span.kind=server system=grpc
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T11:56:19Z" level=info msg="Alloc=27594 TotalAlloc=310840 Sys=46933 NumGC=24 Goroutines=161"
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T12:03:04Z" level=info msg="Notifying 1 settings subscribers: [0xc00055e9a0]"
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T12:03:04Z" level=info msg="Starting configmap/secret informers"cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T12:03:04Z" level=info msg="configmap informer cancelled"
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T12:03:04Z" level=info msg="secrets informer cancelled"
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T12:03:04Z" level=info msg="Configmap/secret informer synced"
cluster-argocd-server-5b64c8449d-dxvrs argocd-server time="2025-06-07T12:06:19Z" level=info msg="Alloc=17459 TotalAlloc=856427 Sys=47189 NumGC=58 Goroutines=163"

In addition to that I've invalidated the cluster cache via the UI and restarted all pods multiple times. I've even re-installed this Argo CD instance but nothing changed. Argo CD is managing the relevant namespace but still not picking app the Argo CD apps even though the permissions have been correctly set up.

Any ideas what I should check next? Atm I'm quite clueless.

Edit:

I've performed some additional checks in order to confirm that lacking privileges aren't the root cause.

❯ kubectl auth can-i get app/self-bootstrapping -n cluster-bootstrap --as=system:serviceaccount:cluster-argocd:cluster-argocd-argocd-application-controller
yes

❯ kubectl auth can-i get app -n cluster-bootstrap --as=system:serviceaccount:cluster-argocd:cluster-argocd-argocd-application-controller
yes

❯ kubectl auth can-i list app -n cluster-bootstrap --as=system:serviceaccount:cluster-argocd:cluster-argocd-argocd-application-controller
yes

❯ kubectl get sa -n cluster-argocd
NAME                                           SECRETS   AGE
cluster-argocd-argocd-application-controller   0         7m12s
cluster-argocd-argocd-redis                    0         7m12s
cluster-argocd-argocd-redis-ha                 0         7m12s
cluster-argocd-argocd-server                   0         7m12s
default     

❯ kubectl auth can-i get ns --as=system:serviceaccount:cluster-argocd:cluster-argocd-argocd-application-controller
Warning: resource 'namespaces' is not namespace scoped

yes

Edit 2:

I've enabled debug logging on the application controller. Deleting and re-applying an app inside of a managed namespace hardly produces any logs.

cluster-argocd-application-controller-0 argocd-application-controller time="2025-06-07T16:31:35Z" level=debug msg="Checking if cluster https://kubernetes.default.svc with clusterShard 0 should be processed by shard 0"
cluster-argocd-application-controller-0 argocd-application-controller time="2025-06-07T16:31:35Z" level=debug msg="Successfully saved info of 1 clusters"
cluster-argocd-application-controller-0 argocd-application-controller time="2025-06-07T16:31:55Z" level=debug msg="Checking if cluster https://kubernetes.default.svc with clusterShard 0 should be processed by shard 0"
cluster-argocd-application-controller-0 argocd-application-controller time="2025-06-07T16:31:55Z" level=debug msg="Successfully saved info of 1 clusters"

[yu-croco]

@elysweyr Since this is about Argo CD itself, you can get more help in upstream.
argo-helm focuses on providing Helm Chart but we don't handle the specific behavior of applications except for Helm install error.