Karpenter picks AMI that is in Pending

[camaeel]

Description

Observed Behavior:

1. I created a ec2nodeClass that picks up nodes that are named with cluster-1-ami-kube-worker-v1.30.8-*
2. I run a packer build of new AMI (custom node class). When the build is still not complete (last message is Waiting for AMI to become ready...), and AMI is in Pending state, the new AMI is picked by Karpenter as the new one.
3. Karpenter detects drift, and creates a new nodeClaim to replace the old "drifted" one

I know I could pin the AMI id, but this would be cumbersome as we rebuild our cluster AMIs frequently.

Expected Behavior:
Karpenter should ignore AMIs that are not in state = Available

Reproduction Steps (Please include YAML):

1. Create node class with amiSelector using name (prefix)
2. create nodepool
3. run some test workload to spin at least one node
4. run packer build with matching ami
5. Observe Karpenter logs and behaviour when AMI is in Pending

Versions:

• Chart Version: 1.1.2
• Kubernetes Version (kubectl version):

Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.30.8

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jmdeal]

Minimally, it seems reasonable for Karpenter to filter out AMIs in a pending state. It's not quite as clear for some of the remaining non-available states, e.g. transient, but more than likely filtering out all states other than available is the correct approach. Until Karpenter supports this, I would recommend pinning either pinning or tagging AMIs once they become available.

[jmdeal]

I was able to verify that available is the only launchable AMI state, we can go forward with a filter.