Unexpected nodeclaim consolidation

[Lily922]

Observed Behavior:
If all instance types of nodepools are unavailable(s.nodeClaimTemplates is empty), an existing nodeclaim will be misjudged as being consolidated. Because all pods on this node were incorrectly determined to be reschedulable to other nodes.

karpenter/pkg/controllers/provisioning/scheduling/scheduler.go

Line 288 in 1db5097

for _, nodeClaimTemplate := range s.nodeClaimTemplates {

If s.nodeClaimTemplates is empty, it means that a new nodeClaim cannot be created and an error should be returned.

[jonathan-innis]

Can you provide the broader context around how you are able to produce this kind of behavior? The code link alone that you shared doesn't indicate to me that there is a bug. What exact inputs and outputs trigger this?

[jonathan-innis]

/triage needs-information

[Lily922]

If all instance types in all existed nodepools are soldout. Existed nodeclaim was deleted abnormally by consolidation even if the pods on this node cannot be rescheduled to other nodes.

In this case, s.nodeClaimTemplates is empty, beause all instance types of nodepools are unavailable.

karpenter/pkg/controllers/provisioning/scheduling/scheduler.go

Line 268 in 1db5097

func (s *Scheduler) add(ctx context.Context, pod *corev1.Pod) error {

will return nil(errs is nil ) because s.nodeClaimTemplates is empty. Return nil means pod (on existing node) will be misjudged as being reschedulable to other nodes/nodeclaims/newnodeclaim. And then all pods of this existing node are misjudged as being reschedulable to other nodes/nodeclaims/newnodeclaim, finally this node will be deleted.

I think maybe add(ctx context.Context, pod *corev1.Pod) should return an error if s.nodeClaimTemplates is empty instead of returning nil.

Like:

func (s *Scheduler) add(ctx context.Context, pod *corev1.Pod) error {
      ......
    // add these codes
    if len(s.nodeClaimTemplates) == 0{
        return fmt.Errorf("all instance types are unavailable")
    }
    // Create new node
    var errs error
    for _, nodeClaimTemplate := range s.nodeClaimTemplates {
        .......
    }
    return errs
  }
}

[k8s-ci-robot]

@jonathan-innis: GitHub didn't allow me to assign the following users: jonathan-inis.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign jonathan-inis

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[jonathan-innis]

/assign jonathan-innis