Skip to content

[BUG] Stale SA token Secrets from K8s < 1.24 survive the crane pipeline #484

Description

@stillalearner

Bug: Stale SA token Secrets from K8s < 1.24 survive the crane pipeline

Summary

When migrating from a Kubernetes cluster < 1.24, auto-generated ServiceAccount token Secrets (kubernetes.io/service-account-token type) pass through the entire crane pipeline (export → transform → apply) and are applied to
the target cluster with invalid source-cluster credentials. The KubernetesPlugin does not whiteout these Secrets because they lack ownerReferences on older K8s versions.

Background

In K8s < 1.24, creating a ServiceAccount automatically generates:

  • A Secret of type kubernetes.io/service-account-token containing a JWT token signed by the source cluster's signing key, the source cluster's CA certificate, and the source SA's UID in annotations
  • These auto-generated Secrets have no ownerReferences on K8s < 1.24

In K8s >= 1.24, these Secrets are no longer auto-generated. Pods use projected tokens via the TokenRequest API instead.

On OpenShift (all versions), kubernetes.io/dockercfg Secrets are also auto-generated per SA with internal registry credentials. On modern OCP (4.16+), these have ownerReferences and are correctly whiteout-ed. On older OCP,
they may not.

Tested Scenario

Source: minikube K8s 1.23 (auto-generated token Secrets, no ownerReferences)
Target: minikube K8s 1.34 (uses projected tokens)

Steps:

# Source cluster (K8s 1.23) has:
# - ServiceAccount "app-sa" with secrets: [{name: app-sa-token-xwzl7}]
# - Secret "app-sa-token-xwzl7" type: kubernetes.io/service-account-token (no ownerReferences)
# - Deployment "myapp" using serviceAccountName: app-sa

crane export --context old-k8s -n app-test
crane transform -e export
crane apply -e export
kubectl --context tgt apply -f output/output.yaml

Current Behavior

Stage Token Secret app-sa-token-xwzl7 SA secrets field
Export Exported as a regular namespaced Secret Exported with secrets: [{name: app-sa-token-xwzl7}]
Transform Survives — no ownerReferences, not in whiteout list Preserved — no SA-specific field stripping
Apply (render) Included in output.yaml with stale token, CA cert, SA UID SA included with dangling Secret reference
kubectl apply (target >= 1.24) Created then immediately deleted by token controller (UID mismatch). No error surfaced. SA created with dangling reference to non-existent Secret
kubectl apply (target < 1.24) Persists with stale source-cluster credentials. Pods get invalid JWT token. SA references Secret with wrong token

Issues on target after migration:

  1. Stale token Secret silently created and deleted: kubectl apply reports secret/app-sa-token-xwzl7 created (exit code 0), but the token controller immediately deletes it because the annotation
    kubernetes.io/service-account.uid doesn't match the target SA's UID. No error is surfaced to the user.
  2. ServiceAccount has dangling Secret reference: The SA is migrated with secrets: [{name: app-sa-token-xwzl7}] pointing to a Secret that no longer exists on the target.
  3. On older target clusters (K8s < 1.24): The stale token Secret would persist. Pods would mount the source cluster's JWT token — signed by the wrong key, with the wrong CA cert. All API calls from the pod would fail
    authentication.

Expected Behavior

  1. Token Secrets should be whiteout-ed during transform. Secrets of type kubernetes.io/service-account-token and kubernetes.io/dockercfg are always cluster-specific and auto-generated — they should never be migrated. The
    target cluster's token controller will generate fresh, valid Secrets for the SA.
  2. SA secrets field should be stripped during transform. The secrets: [{name: app-sa-token-xwzl7}] reference carries over a source-specific Secret name. On the target, this creates a dangling reference. The field should be
    removed so the target SA starts clean.
  3. The migrated app should work cleanly on the target without dangling references, silently deleted resources, or stale credentials. The SA should exist, pods should use the target cluster's token mechanism (projected tokens
    on >= 1.24, auto-generated Secrets on < 1.24), and no source-cluster artifacts should remain.

Metadata

Metadata

Assignees

Labels

kind/bugCategorizes issue or PR as related to a bug.

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
Todo

Relationships

None yet

Development

No branches or pull requests

Issue actions