What happened?
When using AWSClusterStaticIdentity + AWSClusterRoleIdentity with freshly-created IAM access keys (a common pattern for multi-tenant setups), the CAPA controller fails with InvalidClientTokenId due to AWS IAM eventual consistency for new principals.
After the AWS SDK v1 to v2 migration, this transient error becomes persistent because aws.NewCredentialsCache caches the failed result and AWSRolePrincipalTypeProvider.Retrieve() never clears p.credentials after a failure:
// pkg/cloud/identity/identity.go
func (p *AWSRolePrincipalTypeProvider) Retrieve(ctx context.Context) (aws.Credentials, error) {
if p.credentials == nil { // once set, never cleared even on error
// ...
p.credentials = creds
}
return p.credentials.Retrieve(ctx) // cached error persists
}
In SDK v1, credentials.Credentials had IsExpired() which returned true after a failure, causing the next Retrieve() to attempt a fresh AssumeRole. SDK v2's CredentialsCache does not have this behavior -- it caches errors for the cache duration.
What did you expect to happen?
The controller should recover from InvalidClientTokenId on the next reconcile loop, as it did with SDK v1.
How to reproduce
- Create an IAM user + access key
- Immediately create
AWSClusterStaticIdentity + AWSClusterRoleIdentity CRs referencing that key
- CAPA's first
AssumeRole call fails with InvalidClientTokenId
- Subsequent reconcile loops keep failing because the credentials cache is poisoned
Observed behavior
~50 minutes of continuous InvalidClientTokenId errors until KubeadmControlPlane times out. The controller never recovers without a restart.
Environment
- CAPA version: v2.9+ (any version after the SDK v2 migration)
- Kubernetes version: N/A (affects the management cluster controller)
Additional context
CAPA's own e2e test suite already works around this with a time.Sleep(10 * time.Second) after creating access keys (test/e2e/shared/suite.go), acknowledging the propagation delay exists.
Proposed fix
Clear p.credentials when InvalidClientTokenId is returned, so the next reconcile creates a fresh credentials cache:
creds, err := p.credentials.Retrieve(ctx)
if err != nil && isInvalidClientTokenIdError(err) {
p.credentials = nil
}
return creds, err
This restores the SDK v1 behavior with minimal change.
What happened?
When using
AWSClusterStaticIdentity+AWSClusterRoleIdentitywith freshly-created IAM access keys (a common pattern for multi-tenant setups), the CAPA controller fails withInvalidClientTokenIddue to AWS IAM eventual consistency for new principals.After the AWS SDK v1 to v2 migration, this transient error becomes persistent because
aws.NewCredentialsCachecaches the failed result andAWSRolePrincipalTypeProvider.Retrieve()never clearsp.credentialsafter a failure:In SDK v1,
credentials.CredentialshadIsExpired()which returned true after a failure, causing the nextRetrieve()to attempt a freshAssumeRole. SDK v2'sCredentialsCachedoes not have this behavior -- it caches errors for the cache duration.What did you expect to happen?
The controller should recover from
InvalidClientTokenIdon the next reconcile loop, as it did with SDK v1.How to reproduce
AWSClusterStaticIdentity+AWSClusterRoleIdentityCRs referencing that keyAssumeRolecall fails withInvalidClientTokenIdObserved behavior
~50 minutes of continuous
InvalidClientTokenIderrors untilKubeadmControlPlanetimes out. The controller never recovers without a restart.Environment
Additional context
CAPA's own e2e test suite already works around this with a
time.Sleep(10 * time.Second)after creating access keys (test/e2e/shared/suite.go), acknowledging the propagation delay exists.Proposed fix
Clear
p.credentialswhenInvalidClientTokenIdis returned, so the next reconcile creates a fresh credentials cache:This restores the SDK v1 behavior with minimal change.