I found when node comes to the time of consolidateafter, say 10s, every 10s will replace the node with the same shape, seems a bug somewhere

[munger1985]

[wy0824]

@munger1985 Could you share more details to repeat this problem? the nodepool, nodeclass, and the workload's pod yaml.

[munger1985]

if we set consolidateAfter: 30s, whatever nodeclass nodepool will have the new node recreated every 30+ seconds, we are trying to find out why

[munger1985]

ok, we found that is becuz of karpenter is using aws shapes, each shape has a unique cpu and memory set, when it comes to consolidation, aws karpenter 1.4.0 will filter the candidate to be disrupted, which is the oci shape E4.flex, but when define the nodepool, even the shape E4.flex has many cpu settings like 2 vcpus,4 vcpus etc, it will randomly be picked by karpenter 1.4.0, which it found the current node is expensive than the one karpenter picked, karpenter per se will destroy the old node even the new node is with the same shape

[munger1985]

the culprit is vendor>
SIgs.k8s.10 >
karpenter >
pkg

controllers >disruption
types.go
Newcandidate func NewCandidate(ctx context.Context, kubeClient client.Client, recorder events.Recorder, clk clock.Clock, node *state.StateNode, pdbs pdb.Limits,, it will get a candidate before comparing the price,
&Candidate{
StateNode: node.DeepCopy(),
instanceType: instanceType,
NodePool: nodePool,
capacityType: node.Labels()[v1.CapacityTypeLabelKey],
zone: node.Labels()[corev1.LabelTopologyZone],
reschedulablePods: lo.Filter(pods, func(p *corev1.Pod, _ int) bool { return pod.IsReschedulable(p) }),
// We get the disruption cost from all pods in the candidate, not just the reschedulable pods
DisruptionCost: disruptionutils.ReschedulingCost(ctx, pods) * disruptionutils.LifetimeRemaining(clk, nodePool, node.NodeClaim),
}, nil
but this candidate only has shape which is E4.flex, the available shape with cpu settings might be cheaper than this old node, it will terminate old node, create a new node even though the new node is with the same cpu and shape

[wy0824]

with the same shape, different cpu should have different prices, but the memory is not included in the price calculation. I will try to repeat this problem.

[munger1985]

yes, we are from the same team, Jim found this issue

[munger1985]

i have raised pr for karpernter upstream project, if it used the label from nodeclaim(as nodeclaim is created by karpenter not oke), hense more open for downstream project like this one to implement and control . https://github.com/kubernetes-sigs/karpenter/pull/2402/files , after I modified this from karpenter 1.4.0 and some changes in zoom-karpenter, it fixed the issue

[wy0824]

some changes in zoom-karpenter,

could you also share this part?

[munger1985]

sorry for long delay, i missed this msg, i can send u the fix, becuz it is based on karpenter sigs changes, so i did not push the commit here in this repo . will make a new branch for your reference.

[munger1985]

@wy0824 https://github.com/munger1985/karpenter-oci.git disruptissue branch, and the bugfix for karpenter sigs is only 2 lines of code, it is in https://github.com/kubernetes-sigs/karpenter/pull/2402/files , the least change to karpenter sigs

[munger1985]

these are changes in karpenter-oci

[munger1985]

@wy0824 I guess why it takes so long for karpentersigs to merge pr is becuz the changes to them is too much, hence my fix, my fix just 2 lines changes to karpentersigs , and as long as we can control the value of the tag from karpenter sigs, I can control everything in karpenter oci, so pls test my https://github.com/munger1985/karpenter-oci.git disruptissue branch, in case of delayed response, pls contact me jingsong.liu@oracle.com

[wy0824]

checked your commit in karpenter-oci, your thought is to change the instance name with the cpu and mem info. We considered this way before, this may take a break change on the customer side, like before the change, configure the E4.Flex in nodepool, but after that, we need to change to E4.Flex-2-1024, E4.Flex-2-2048. We need to discuss in our team

[munger1985]

yes,exactly, the most difficult part to me is how to push karpenter sigs merges as simple as possible, once karpenter sigs has updated this, we can do whatever we want in oci implementation -- karpenter-oci