[Bug]: Provider ignores ServiceAccount annotation and uses mounted SA instead, causing 403 errors

[fahedouch]

Is there an existing issue for this?

• I have searched the existing issues

Affected Resource(s)

Workload Identity Federation Crossplane resources

Resource MRs required to reproduce the bug

No response

Steps to Reproduce

I used this documentation setup workload identity for my GKE cluster

What happened?

I'm encountering a 403 error in Crossplane logs when using the GCP provider. The issue appears to be related to Workload Identity configuration.

Problem

The provider is ignoring the service account specified via the iam.gke.io/gcp-service-account annotation:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: crossplane-gar-sa@project-id.iam.gserviceaccount.com
  name: crossplane
  namespace: crossplane-system

Instead, it uses the provider's mounted service account (via Application Default Credentials), which causes permission errors.

Current Workaround

The issue is resolved by granting the following permissions to the provider's service account:

# Grant workloadIdentityUser role
gcloud iam service-accounts add-iam-policy-binding sa-crossplane@xxx-staging.iam.gserviceaccount.com \
  --role=roles/iam.workloadIdentityUser \
  --member="principal://iam.googleapis.com/projects/XXXXXX/locations/global/workloadIdentityPools/xxx-staging.svc.id.goog/subject/system:serviceaccount:crossplane-system:*" \
  --project=xxx-staging

# Grant serviceAccountTokenCreator role
gcloud iam service-accounts add-iam-policy-binding sa-crossplane@xxx-staging.iam.gserviceaccount.com \
  --role=roles/iam.serviceAccountTokenCreator \
  --member="principal://iam.googleapis.com/projects/XXXXXX/locations/global/workloadIdentityPools/xxx-staging.svc.id.goog/subject/system:serviceaccount:crossplane-system:*" \
  --project=xxx-staging

Root Cause

After reviewing the provider codebase, I found:

// We don't need to do anything here, as the TF Provider will take care of workloadIdentity etc.

The provider delegates authentication to the Terraform GCP provider, which uses Application Default Credentials and the pod's mounted service account rather than respecting the Kubernetes ServiceAccount annotation.

Questions

1. Should I follow the official Crossplane Workload Identity documentation that recommends using
   ServiceAccount annotations?
2. Or should I use InjectedIdentity with direct IAM bindings instead:

apiVersion: gcp.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: xxx-staging
spec:
  credentials:
    source: InjectedIdentity
  projectID: xxx-staging

Relevant Error Output Snippet

Crossplane Version

xpkg.crossplane.io/crossplane/crossplane:v2.2.0

Provider Version

xpkg.upbound.io/upbound/provider-gcp-cloudplatform:v2.4.0

Kubernetes Version

No response

Kubernetes Distribution

No response

Additional Info

No response