Cloud Controller Manager fails to discover new VMs after upgrading to Nutanix Cloud Provider v0.6.0

[g0k8s]

/kind bug

What steps did you take and what happened:

After upgrading to Nutanix Cloud Provider v0.6.0, the cloud-controller-manager no longer discovers newly added VMs. This results in nodes either being continuously recreated or remaining tainted and unschedulable.

Issue 1: VM repeatedly recreated / VM not found

After provisioning a VM and joining it to the Kubernetes cluster, the node keeps getting recreated. The following logs are observed in the nutanix-cloud-controller-manager pod:

I1218 15:26:11.890746       1 node_controller.go:429] Initializing node devk8s01-md-worker-default-jz9kk-khkw5 with cloud provider
2025-12-18 15:26:11.891Z INFO - GET https://10.10.201.20:9440/api/vmm/v4.1/ahv/config/vms/3fbf7e7f-509d-4612-b58a-c28eb2f5fc39
2025-12-18 15:26:11.906Z INFO - HTTP/1.1 404 NOT FOUND
I1218 15:26:11.906873       1 node_controller.go:233] error syncing 'devk8s01-md-worker-default-jz9kk-khkw5': failed to get instance metadata for node devk8s01-md-worker-default-jz9kk-khkw5: failed to get VM: API call failed: {"data":{"error":[{"message":"Failed to perform the operation on the VM with UUID '3fbf7e7f-509d-4612-b58a-c28eb2f5fc39', because it is not found.","severity":"ERROR","code":"VMM-30100","locale":"en-US","errorGroup":"VM_NOT_FOUND","argumentsMap":{"vm_uuid":"3fbf7e7f-509d-4612-b58a-c28eb2f5fc39"},"$objectType":"vmm.v4.error.AppMessage"}],"$errorItemDiscriminator":"List<vmm.v4.error.AppMessage>","$objectType":"vmm.v4.error.ErrorResponse"},"$dataItemDiscriminator":"vmm.v4.error.ErrorResponse"}, requeuing
E1218 15:26:11.906912       1 node_controller.go:244] "Unhandled Error" err="error syncing 'devk8s01-md-worker-default-jz9kk-khkw5': failed to get instance metadata for node devk8s01-md-worker-default-jz9kk-khkw5: failed to get VM: API call failed: {...}, requeuing" logger="UnhandledError"

The VM exists and is visible in Prism, but the CCM fails to retrieve it via the API and continuously retries.

Issue 2: VM discovered but node remains tainted

On another cluster, the VM is successfully found, but the node never becomes initialized and remains tainted:

2025-12-23 15:09:46.712Z INFO - GET https://10.10.201.20:9440/api/vmm/v4.1/ahv/config/vms/74d90883-9a67-461c-5a75-f8070d9a6470
2025-12-23 15:09:46.736Z INFO - HTTP/1.1 200 OK
I1223 15:09:46.738543       1 node_controller.go:233] error syncing 'pock8s01-md-worker-default-kxh9x-wl2wt': failed to get instance metadata for node pock8s01-md-worker-default-kxh9x-wl2wt: unable to determine network interfaces from VM with UUID 74d90883-9a67-461c-5a75-f8070d9a6470, requeuing

The node remains tainted indefinitely:

Taints:
  node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
  node.cluster.x-k8s.io/uninitialized:NoSchedule
Unschedulable: false

What did you expect to happen:

New nodes are added to the cluster and the "unschedulable" taints are removed.

Anything else you would like to add:

VMs are created from a machine template hosted on prism central.
Workaround: Downgrading to a previous Nutanix Cloud Provider version restores normal behavior.

Environment:

• prism central version: 7.3
• cloud-provider-nutanix version: v0.6.0
• Kubernetes version: (use kubectl version): v1.33.5
• OS (e.g. from /etc/os-release): Ubuntu 22.04

[tuxtof]

@g0k8s what it your k8s distribution/install method ?

[g0k8s]

We use kubernetes image-builder in order to generate a VM image, these newly provisionned image are then added to the cluster as a node by kubernetes cluster-api

[cjpluss]

We have experienced the same issue, via Nutanix NKP.

NKP v2.17.0
Prism Central pc.7.3.1.3
CCM v0.6.0
Kubernetes v1.34.1
Node Image nkp-rocky-9.6-release-cis-1.34.1-20251206060914

Behaviour seen upon cluster upgrade and expansion of node pools (ie any point VMs are created). The VM is created successfully and obtains an IP address. It is added to the Kubernetes cluster but never obtains an INTERNAL-IP:

$ kubectl get node -o wide
NAME                                 STATUS   ROLES           AGE    VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                      KERNEL-VERSION                 CONTAINER-RUNTIME
...
poc-cluster-md-0-g7zg7-rptcc-zmvdc   Ready    <none>          38m    v1.34.1   <none>         <none>        Rocky Linux 9.6 (Blue Onyx)   5.14.0-570.18.1.el9_6.x86_64   containerd://1.7.29-d2iq.1

A nutanixmachine object exists on the management cluster and has the correct IP:

$ kubectl get nutanixmachine -A
NAMESPACE                 NAME                                 ADDRESS        READY   PROVIDERID                                       FAILUREDOMAIN
...
poc-cluster-2lbmk-g5w2t   poc-cluster-md-0-g7zg7-rptcc-zmvdc   172.20.11.72   true    nutanix://09035ba8-4357-4c33-7462-fd5b79e46d93   

CCM logs show the issue:

2026-01-27 12:29:12.586Z INFO - GET https://redacted.address:9440/api/vmm/v4.1/ahv/config/vms/09035ba8-4357-4c33-7462-fd5b79e46d93
2026-01-27 12:29:12.679Z INFO - HTTP/1.1 200 OK
I0127 12:29:12.681228       1 node_controller.go:233] error syncing 'poc-cluster-md-0-g7zg7-rptcc-zmvdc': failed to get instance metadata for node poc-cluster-md-0-g7zg7-rptcc-zmvdc: unable to determine network interfaces from VM with UUID 09035ba8-4357-4c33-7462-fd5b79e46d93, requeuing
E0127 12:29:12.681294       1 node_controller.go:244] "Unhandled Error" err="error syncing 'poc-cluster-md-0-g7zg7-rptcc-zmvdc': failed to get instance metadata for node poc-cluster-md-0-g7zg7-rptcc-zmvdc: unable to determine network interfaces from VM with UUID 09035ba8-4357-4c33-7462-fd5b79e46d93, requeuing" logger="UnhandledError"

Curl to Nutanix API using the same credentials CCM is using show an IP address:

$ curl -sk -u $PC_AUTH "https://redacted.address:9440/api/vmm/v4.1/ahv/config/vms/${VM_UUID}"   | jq '.data | {name:.name, uuid:.extId, nics:.nics}'
{
  "name": "poc-cluster-md-0-g7zg7-rptcc-zmvdc",
  "uuid": "09035ba8-4357-4c33-7462-fd5b79e46d93",
  "nics": [
    {
      "$reserved": {
        "$fv": "v4.r1"
      },
      "$objectType": "vmm.v4.ahv.config.Nic",
      "extId": "01962e2f-2578-4e58-a94b-04efc437691c",
      "backingInfo": {
        "$reserved": {
          "$fv": "v4.r1"
        },
        "$objectType": "vmm.v4.ahv.config.EmulatedNic",
        "macAddress": "50:6b:8d:d4:e5:5b",
        "isConnected": true,
        "numQueues": 1
      },
      "networkInfo": {
        "$reserved": {
          "$fv": "v4.r1"
        },
        "$objectType": "vmm.v4.ahv.config.NicNetworkInfo",
        "nicType": "NORMAL_NIC",
        "subnet": {
          "$reserved": {
            "$fv": "v4.r1"
          },
          "$objectType": "vmm.v4.ahv.config.SubnetReference",
          "extId": "d3d07af6-53ea-4899-8f51-6ee402490153"
        },
        "vlanMode": "ACCESS",
        "ipv4Config": {
          "$reserved": {
            "$fv": "v4.r1"
          },
          "$objectType": "vmm.v4.ahv.config.Ipv4Config",
          "ipAddress": {
            "$reserved": {
              "$fv": "v1.r0"
            },
            "$objectType": "common.v1.config.IPv4Address",
            "value": "172.20.11.72",
            "prefixLength": 32
          }
        }
      }
    }
  ]
}

We had no such issue with NKP 2.16 - presumably used a previous version of CCM. I have an open ticket with Nutanix Support but thought I'd share this here also.

[cjpluss]

Follow up from my interaction with support; they said it's a known issue with AOS 7.0 & PC 7.3. It is actually specified that AOS 7.3 is the minimum for NKP 2.17 so I'll upgrade and see.

[thunderboltsid]

Nutanix Cloud Controller Manager (CCM) v0.6.0 uses the newer Prism v4 APIs. These APIs require Prism Central 7.3 or later and AOS 7.3 or later.

If your environment is running earlier Prism Central or AOS versions, CCM v0.6.0 may not function correctly. In such cases, please continue using a previous CCM release until the infrastructure is upgraded to the supported versions.