Skip to content

AWSRolePrincipalTypeProvider caches InvalidClientTokenId error after SDK v2 migration, preventing recovery on next reconcile #6123

Description

@omargallob

What happened?

When using AWSClusterStaticIdentity + AWSClusterRoleIdentity with freshly-created IAM access keys (a common pattern for multi-tenant setups), the CAPA controller fails with InvalidClientTokenId due to AWS IAM eventual consistency for new principals.

After the AWS SDK v1 to v2 migration, this transient error becomes persistent because aws.NewCredentialsCache caches the failed result and AWSRolePrincipalTypeProvider.Retrieve() never clears p.credentials after a failure:

// pkg/cloud/identity/identity.go
func (p *AWSRolePrincipalTypeProvider) Retrieve(ctx context.Context) (aws.Credentials, error) {
    if p.credentials == nil {  // once set, never cleared even on error
        // ...
        p.credentials = creds
    }
    return p.credentials.Retrieve(ctx)  // cached error persists
}

In SDK v1, credentials.Credentials had IsExpired() which returned true after a failure, causing the next Retrieve() to attempt a fresh AssumeRole. SDK v2's CredentialsCache does not have this behavior -- it caches errors for the cache duration.

What did you expect to happen?

The controller should recover from InvalidClientTokenId on the next reconcile loop, as it did with SDK v1.

How to reproduce

  1. Create an IAM user + access key
  2. Immediately create AWSClusterStaticIdentity + AWSClusterRoleIdentity CRs referencing that key
  3. CAPA's first AssumeRole call fails with InvalidClientTokenId
  4. Subsequent reconcile loops keep failing because the credentials cache is poisoned

Observed behavior

~50 minutes of continuous InvalidClientTokenId errors until KubeadmControlPlane times out. The controller never recovers without a restart.

Environment

  • CAPA version: v2.9+ (any version after the SDK v2 migration)
  • Kubernetes version: N/A (affects the management cluster controller)

Additional context

CAPA's own e2e test suite already works around this with a time.Sleep(10 * time.Second) after creating access keys (test/e2e/shared/suite.go), acknowledging the propagation delay exists.

Proposed fix

Clear p.credentials when InvalidClientTokenId is returned, so the next reconcile creates a fresh credentials cache:

creds, err := p.credentials.Retrieve(ctx)
if err != nil && isInvalidClientTokenIdError(err) {
    p.credentials = nil
}
return creds, err

This restores the SDK v1 behavior with minimal change.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/provider/awsIssues or PRs related to aws providerkind/bugCategorizes issue or PR as related to a bug.needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions