Unexpected nodeclaim consolidation

[Lily922]

Observed Behavior:
If all instance types of nodepools are unavailable(s.nodeClaimTemplates is empty), an existing nodeclaim will be misjudged as being consolidated. Because all pods on this node were incorrectly determined to be reschedulable to other nodes.

karpenter/pkg/controllers/provisioning/scheduling/scheduler.go

Line 288 in 1db5097

for _, nodeClaimTemplate := range s.nodeClaimTemplates {

If s.nodeClaimTemplates is empty, it means that a new nodeClaim cannot be created and an error should be returned.

[jonathan-innis]

Can you provide the broader context around how you are able to produce this kind of behavior? The code link alone that you shared doesn't indicate to me that there is a bug. What exact inputs and outputs trigger this?

[jonathan-innis]

/triage needs-information

[Lily922]

If all instance types in all existed nodepools are soldout. Existed nodeclaim was deleted abnormally by consolidation even if the pods on this node cannot be rescheduled to other nodes.

In this case, s.nodeClaimTemplates is empty, beause all instance types of nodepools are unavailable.

karpenter/pkg/controllers/provisioning/scheduling/scheduler.go

Line 268 in 1db5097

func (s *Scheduler) add(ctx context.Context, pod *corev1.Pod) error {

will return nil(errs is nil ) because s.nodeClaimTemplates is empty. Return nil means pod (on existing node) will be misjudged as being reschedulable to other nodes/nodeclaims/newnodeclaim. And then all pods of this existing node are misjudged as being reschedulable to other nodes/nodeclaims/newnodeclaim, finally this node will be deleted.

I think maybe add(ctx context.Context, pod *corev1.Pod) should return an error if s.nodeClaimTemplates is empty instead of returning nil.

Like:

func (s *Scheduler) add(ctx context.Context, pod *corev1.Pod) error {
      ......
    // add these codes
    if len(s.nodeClaimTemplates) == 0{
        return fmt.Errorf("all instance types are unavailable")
    }
    // Create new node
    var errs error
    for _, nodeClaimTemplate := range s.nodeClaimTemplates {
        .......
    }
    return errs
  }
}