Karpenter creates hundreds of InstanceProfiles

[varun-armis]

Description

Observed Behavior:

Karpenter will create hundreds of "zombie" instance profiles that caused us to reach our AWS Quota and we started seeing reconciler errors

{"level":"ERROR","time":"2025-12-31T08:27:43.178Z","logger":"controller","caller":"controller/controller.go:474","message":"Reconciler error","commit":"c9c3a48","controller":"nodeclass","controllerGroup":"karpenter.k8s.aws","controllerKind":"EC2NodeClass","EC2NodeClass":{"name":"sandbox-x86"},"namespace":"","name":"sandbox-x86","reconcileID":"daa53858-4bd7-4456-848b-29345886c3de","aws-operation-name":"CreateInstanceProfile","aws-request-id":"9d5f627e-3e35-4711-9021-c9f5b │ │ e5746d7","aws-service-name":"IAM","aws-status-code":409,"instance-profile":"frstg2_4574861420783301831","error":"creating instance profile, operation error IAM: CreateInstanceProfile, https response error StatusCode: 409, RequestID: 9d5f627e-3e35-4711-9021-c9f5be5746d7, LimitExceeded: Cannot exceed quota for InstanceProfilesPerAccount: 1000 (aws-operation-name=CreateInstanceProfile, aws-request-id=9d5f627e-3e35-4711-9021-c9f5be5746d7, aws-service-name=IAM, aws-status-cod │ │ e=409) (instance-profile=frstg2_4574861420783301831)"}

We realized that we had 2 misconfigurations. First, we were accidentally applying resource filtering on our ListIamInstanceProfiles policy (which is not supported: https://docs.aws.amazon.com/service-authorization/latest/reference/list_awsidentityandaccessmanagementiam.html) that was attached to the Karpenter controller. Second, we were also using the IAM Role ARN instead of name in all of our EC2NodeClasses

Expected Behavior:

According to this (instanceprofile.go) it seems like using the ARN of the IAM Role in the EC2NodeClass shouldn't actually be supported yet we've been running Karpenter with the Role ARN for several months. It seems like the fix should either be to support the ARN or display an error when the ARN is used for the role.

Reproduction Steps (Please include YAML):

1. Set the role for the EC2NodeClass to the ARN of the IAM Role instead of the name of the IAM Role.
2. Do not attach the ListIamInstanceProfiles permission to the Karpenter controller

Versions:

• Chart Version: 1.8.3
• Kubernetes Version (kubectl version): 1.33

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jigisha620]

@varun-armis ,
So if I understand correctly the issue that you were running into which is the instanceProfiles not getting cleaned up was because of misconfigured permission. Once the ListIamInstanceProfiles permissions is granted correctly, these profiles should get cleaned up by Karpenter if they are no longer being used. I believe you were already able to fix that, right?

According to this (instanceprofile.go) it seems like using the ARN of the IAM Role in the EC2NodeClass shouldn't actually be supported yet we've been running Karpenter with the Role ARN for several months. It seems like the fix should either be to support the ARN or display an error when the ARN is used for the role.

Using IAM role ARN is already supported. This is where we get the roleName from the ARN in the code - https://github.com/aws/karpenter-provider-aws/blob/main/pkg/providers/instanceprofile/instanceprofile.go#L166. So it should work.

[varun-armis]

So we actually observed that after fixing the ListIamInstanceProfiles permissions in the Karpenter controller, we additionally had to update the IAM role from ARN to Name to order for the InstanceProfiles to get cleaned up which makes it seem like IAM Role ARN isn't fully supported

[jigisha620]

@varun-armis ,
That shouldn't happen because once Karpenter Controller is able to list the InstanceProfiles, it should be able to see the role associated with it and as part of the cleanup it should be able remove role from InstanceProfile and then perform the delete. I have tried to reproduce the issue on my end by using a roleArn instead of the name and it cleaned up the instanceProfile. That being said, can you describe the instanceProfiles (a few of them) to see what role they are associated with? What's the roleArn on your ec2nodeclass? What did you change it to? Also can you share karpenter controller logs from when this happened?

[alexeiser]

FWIW - I had a related issue using a role ARN - it’s not that the cleanup didn’t happen - it’s that it happened every few minutes. Switching to the name - stopped the excessive instance profile creation events.

[varun-armis]

All the instanceProfiles were associated with the same exact role which was just our <cluster_name>-workers IAM role. We're running in AWS GovCloud. Could that be a factor here?

Example:
InstanceProfile: arn:aws-us-gov:iam::<aws_account_id>:instance-profile/karpenter/us-gov-east-1/<cluster_name>/c7a5c5f8-****/<cluster_name>_554751
Role ARN: arn:aws-us-gov:iam::<aws_account_id>:role/<cluster_name>-workers
Role Name: <cluster_name>-workers

Change: Role ARN -> Role Name

Here's an example of the controller log that we got

│ {"level":"ERROR","time":"2025-12-31T08:27:43.178Z","logger":"controller","caller":"controller/controller.go:474","message":"Reconciler error","commit":"c9c3a48","controller":"nodeclass","controllerGroup":"karpenter.k8s.aws","controllerKind":"EC2NodeClass","EC2NodeClass":{"name":"sandbox-x86"},"namespace":"","name":"sandbox-x86","reconcileID":"daa53858-4bd7-4456-848b-29345886c3de","aws-operation-name":"CreateInstanceProfile","aws-request-id":"9d5f627e-3e35-4711-9021-c9f5b │ │ e5746d7","aws-service-name":"IAM","aws-status-code":409,"instance-profile":"frstg2_4574861420783301831","error":"creating instance profile, operation error IAM: CreateInstanceProfile, https response error StatusCode: 409, RequestID: 9d5f627e-3e35-4711-9021-c9f5be5746d7, LimitExceeded: Cannot exceed quota for InstanceProfilesPerAccount: 1000 (aws-operation-name=CreateInstanceProfile, aws-request-id=9d5f627e-3e35-4711-9021-c9f5be5746d7, aws-service-name=IAM, aws-status-cod │ │ e=409) (instance-profile=frstg2_4574861420783301831)"}

[saurav-agarwalla]

Karpenter's EC2NodeClass doesn't support ARNs for the IAM role. It only supports only the role name today. For some reason, it looks like we don't clarify in the docs that role ARNs aren't supported. Let me see if I can add support for ARNs since I think it shouldn't be a huge change.