Machinedeployment replicas not in sync with Machineset

[KTAN119]

What steps did you take and what happened?

When I delete a MD. It won't change the phase of MD to scaling down but stuck at Running. And the replicas, is not in sync with Machineset. As cluster autoscaler is monitoring the replicas of MD, it will cause delay in scaling up of other nodegroup.

What did you expect to happen?

When deleting a MD, it will change the phase of MD to scaling down. And the replicas need to be in sync with Machineset

Cluster API version

v1.9.6

Kubernetes version

v1.31.4+rke2r1

Anything else you would like to add?

No response

Label(s) to be applied

/kind bug
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPI contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[enxebre]

Is this because we implemented foregraound deletion? Should autoscaler discard scalable resources with a deletion timestamp from nodeGroups()?

[KTAN119]

Hi @enxebre . Currently autoscaler make decision based on the replicas shown in machinedeployment. Can we sync the machinedeployment status and machineset status? As I believe this is some kind of bug.

[enxebre]

As we implemented foreground deletion I assume we impacted capi autoscaler since for capi autoscaler we discover scalable resources i.e. machineDeployment. Before foreground the machineDeployment will be removed right away preventing the autoscaler from discovering it. After foreground the resource remains in etcd until the operation is complete hence the. behaviour you are seeing.
So regardless of the MachineDeployment refreshing replicas in status during a delete operation which sounds reasonable to me and is done for v1beta2, we should validate if it make sense for autoscaler to not discover scalable resources with a deletion timestamp as nodegroups, which would basically preserver the behaviour before implementing foreground

[KTAN119]

Hi @enxebre. I think it make sense for autoscaler to still discover resources with deletion timestamp.

For example, in the case where the deletion of 100+nodegroup1, if autoscaler detected there is a deletion event of this 100+ nodegroup1 and directly scale up 100+nodegroup2, it dont make sense. It should scale up 1 nodegroup2 after the deletion of 1 nodegroup1.

[sbueringer]

Makes sense to me in general.

I think I would only do it as part of the v1beta2 work though as updating the top-level .status replica counters will require some refactoring in the MD controller (which I think we'll have to do anyway for v1beta2).

Regarding the Phase, not sure if it should be ScalingDown or Deleting. We use Deleting in the Cluster CR.

I'll add this issue to the v1beta2 umbrella issue