Unable to create an Argo CD Application using Google Cloud Workload Identity Federation for GKE

[KeitaroSaito]

⚠️Sorry, I am Japanese. English is not good🙇

What I’m struggling with

I want to connect Argo CD and the product application—each running in different Google Cloud projects and GKE clusters—via Workload Identity, but it isn’t working.

Background

I followed the official Argo CD and Google Cloud references while asking ChatGPT for guidance.
I connect to Argo CD through port-forwarding.

Steps I’ve taken

Set up separate Google Cloud projects and GKE clusters for Argo CD and the application

• project-a: project that hosts Argo CD
• project-b: project that deploys the application
• Created GKE cluster argocd in project-a
• Created GKE cluster prod in project-b

Prepare a GSA for external workload access

Create the GSA in project-b

gcloud iam service-accounts create argocd-connect \
    --project=project-b

Grant roles to the GSA

gcloud projects add-iam-policy-binding project-b \
 --member="serviceAccount:[email protected]" \
 --role="roles/container.clusterViewer"

gcloud projects add-iam-policy-binding project-b \
  --member="serviceAccount:[email protected]" \
  --role="roles/container.viewer"

gcloud projects add-iam-policy-binding project-b \
 --member="serviceAccount:[email protected]" \
 --role="roles/container.developer"

gcloud projects add-iam-policy-binding project-b \
 --member="serviceAccount:[email protected]" \
 --role="roles/iam.workloadIdentityUser"

gcloud projects add-iam-policy-binding project-b \
  --member="serviceAccount:[email protected]" \
  --role="roles/iam.serviceAccountTokenCreator"

Install Argo CD

# On the cluster for Argo CD
kubectl create namespace argocd

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Configure each KSA in Argo CD

Add iam-policy-binding for each KSA

gcloud iam service-accounts add-iam-policy-binding [email protected] \
 --project=project-b \
 --role=roles/iam.workloadIdentityUser \
 --member="serviceAccount:project-a.svc.id.goog[argocd/argocd-application-controller]"

gcloud iam service-accounts add-iam-policy-binding [email protected] \
 --project=project-b \
 --role=roles/iam.workloadIdentityUser \
 --member="serviceAccount:project-a.svc.id.goog[argocd/argocd-repo-server]"

gcloud iam service-accounts add-iam-policy-binding [email protected] \
 --project=project-b \
 --role=roles/iam.workloadIdentityUser \
 --member="serviceAccount:project-a.svc.id.goog[argocd/argocd-server]"

Add annotations to each KSA

kubectl annotate serviceaccount argocd-application-controller \
 -n argocd \
 iam.gke.io/gcp-service-account=argocd-connect@project-b.iam.gserviceaccount.com \
 --overwrite

kubectl annotate serviceaccount argocd-repo-server \
 -n argocd \
 iam.gke.io/gcp-service-account=argocd-connect@project-b.iam.gserviceaccount.com \
 --overwrite

kubectl annotate serviceaccount argocd-server \
 -n argocd \
 iam.gke.io/gcp-service-account=argocd-connect@project-b.iam.gserviceaccount.com \
 --overwrite

Make Argo CD recognize the target cluster

Customize the Argo CD image to include commands required for Workload Identity

According to ChatGPT, because we use gke-gcloud-auth-plugin, we need to install the related commands.

FROM quay.io/argoproj/argocd:latest

USER 0

# Install gcloud SDK & gke-gcloud-auth-plugin
RUN apt-get update && apt-get install -y curl apt-transport-https ca-certificates gnupg \
  && echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" \
    > /etc/apt/sources.list.d/google-cloud-sdk.list \
  && curl -sSL https://packages.cloud.google.com/apt/doc/apt-key.gpg \
    | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg \
  && apt-get update && apt-get install -y google-cloud-sdk \
  && apt-get install -y google-cloud-sdk-gke-gcloud-auth-plugin \
  && apt-get clean

# Install kubectl
RUN curl -LO "https://dl.k8s.io/release/$(curl -sL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" \
  && install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl \
  && rm kubectl

# Add self-signed certificate
COPY my-cluster.crt /usr/local/share/ca-certificates/my-cluster.crt
RUN update-ca-certificates

ENV PATH="$PATH:/usr/lib/google-cloud-sdk/bin"

USER 999

ENV USE_GKE_GCLOUD_AUTH_PLUGIN=True

docker buildx build \
  --platform linux/amd64 \
  -t asia-docker.pkg.dev/project-a/argocd-custom/argocd-with-gcloud:latest \
  --push .

Create the Cluster Secret

# mycluster-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mycluster-secret
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: gb-cluster
  server: https://GKE.public.endpoint
  config: |
    {
      "apiVersion": "v1",
      "kind": "Config",
      "clusters": [{
        "name": "aut-clus",
        "cluster": {
          "server": "https://GKE.public.endpoint",
          "tlsClientConfig": {
            "insecure": false,
            "caData": "LS0tLS1CRU..."
          }
        }
      }],
      "contexts": [{
        "name": "aut-clus",
        "context": {
          "cluster": "aut-clus",
          "user": "argocd-gsa"
        }
      }],
      "current-context": "aut-clus",
      "users": [{
        "name": "argocd-gsa",
        "user": {
          "exec": {
            "apiVersion": "client.authentication.k8s.io/v1beta1",
            "command": "gke-gcloud-auth-plugin",
            "args": ["--impersonate_service_account=argocd-connect@project-b.iam.gserviceaccount.com"],
            "interactiveMode": "Never"
          }
        }
      }]
    }

kubectl -n argocd apply -f mycluster-secret.yaml

With this, https://<GKE public endpoint> appears in Argo CD, but its status is unknown.

Create the Application

argocd app create guestbook-app \
 --repo https://github.com/KeitaroSaito/argocd-example-apps.git \
 --path guestbook \
 --dest-server https://<GKE public endpoint> \
 --dest-namespace prod \
 --directory-recurse \
 --sync-policy automated \
 --grpc-web \
 --loglevel debug

This command fails with the following error:

{
  "level": "fatal",
  "msg": "rpc error: code = PermissionDenied desc = error while validating and normalizing app: error validating the repo: error getting API resources: failed to discover server resources, zero resources returned: unknown",
  "time": "2025-06-23T16:13:07+09:00"
}

Additional information

If I enter the argocd-server pod, create a kubeconfig from stringData.config in mycluster-secret.yaml, and use kubectl, I can retrieve information from the target cluster—so I believe Workload Identity is configured correctly.
Therefore, I think the issue lies with argocd app create failing for some other reason.

kubectl -n argocd exec -it deployment/argocd-server -- bash

cat <<EOF > /tmp/kubeconfig.json
{
    "apiVersion": "v1",
    "kind": "Config",
    "clusters": [{
        "name": "aut-clus",
        "cluster": {
            "server": "https://GKE.public.endpoint",
            "tlsClientConfig": {
                "insecure": false,
                "caData": "LS0tLS1CRU..."
            }
        }
    }],
    "contexts": [{
        "name": "aut-clus",
        "context": {
            "cluster": "aut-clus",
            "user": "argocd-gsa"
        }
    }],
    "current-context": "aut-clus",
    "users": [{
        "name": "argocd-gsa",
        "user": {
            "exec": {
                "apiVersion": "client.authentication.k8s.io/v1beta1",
                "command": "gke-gcloud-auth-plugin",
                "args": ["--impersonate_service_account=argocd-connect@project-b.iam.gserviceaccount.com"],
                "interactiveMode": "Never"
            }
        }
    }]
}
EOF

kubectl get ns --kubeconfig=/tmp/kubeconfig

Namespace available