Deploy control-plane across multiple vCenters (vCenter Enhanced Linked Mode)

[jisnardo]

Hi, I was able to deploy worker nodes across 3 different vcenters but I have not foud the way to set different server for CP.

Could be possible?

https://cloud-provider-vsphere.sigs.k8s.io/tutorials/deploying_cpi_with_multi_dc_vc_aka_zones

Imagine this:

vsphere1 (location1)

master1
master3
worker-vsphere1
worker-vsphere1
worker-vsphere1

vsphere2 (location2)

master2 ---> (location1, expected location2)
worker-vsphere2
worker-vsphere2
worker-vsphere2

vsphere3 (location3)

master4 ---> (location1, expected location3)
master5 ---> (location1, expected location3)
workers-vsphere3
workers-vsphere3
workers-vsphere3

VSphereDeploymentZone and VSphereFailureDomain like this:

- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: VSphereFailureDomain
  metadata:
    name: location1
  spec:
    region:
      autoConfigure: false
      name: location1
      tagCategory: k8s-region
      type: Datacenter
    topology:
      computeCluster: LOCATION1
      datacenter: LOCATION1
      datastore: LOCATION1
      networks:
      - VLAN_NET
    zone:
      autoConfigure: false
      name: location1
      tagCategory: k8s-zone
      type: ComputeCluster

- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: VSphereFailureDomain
  metadata:
    name: location2
  spec:
    region:
      autoConfigure: false
      name: location2
      tagCategory: k8s-region
      type: Datacenter
    topology:
      computeCluster: LOCATION2
      datacenter: LOCATION2
      datastore: LOCATION2
      networks:
      - VLAN_NET
    zone:
      autoConfigure: false
      name: location2
      tagCategory: k8s-zone
      type: ComputeCluster
...

- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: VSphereDeploymentZone
  metadata:
    name: location1
  spec:
    controlPlane: true
    failureDomain: location1
    placementConstraint:
      resourcePool: LOCATION1/Resources
    server: location1.test.com

- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: VSphereDeploymentZone
  metadata:
    name: location2
  spec:
    controlPlane: true
    failureDomain: location2
    placementConstraint:
      resourcePool: LOCATION2/Resources
    server: location2.test.com
...

VSphereCluster has failureDomainSelector: {}

vsphere.conf like this:

https://cloud-provider-vsphere.sigs.k8s.io/cloud_config

apiVersion: v1
data:
  vsphere.conf: |
    global:
      port: 443
      insecureFlag: true
    vcenter:
      location1.test.com:
        secretName: cloud-provider-credentials-location1
        secretNamespace: kube-system
        datacenters:
        - 'LOCATION1'
        server: 'location1.test.com'
      location2.test.com:
        secretName: cloud-provider-credentials-location2
        secretNamespace: kube-system
        datacenters:
        - 'LOCATION02'
        server: 'location2.test.com'
    labels:
      region: k8s-region
      zone: k8s-zone
kind: ConfigMap
metadata:
  name: cloud-config
  namespace: kube-system

Regards.

[chrischdi]

Could you maybe show how you make it work for the MachineDeployments?

The full yaml examples for MachineDeployments + VSphereMachineTemplate + KubeadmControlPlane would be appreciated.

[jisnardo]

Hi @chrischdi, here it is.

---
apiVersion: ipam.cluster.x-k8s.io/v1alpha2
kind: InClusterIPPool
metadata:
  name: capvrke2m
  namespace: capvrke2
spec:
  prefix: 16
  gateway: 172.16.0.1
  addresses:
    - 172.16.1.1-172.16.1.30
---
apiVersion: ipam.cluster.x-k8s.io/v1alpha2
kind: InClusterIPPool
metadata:
  name: capvrke2wa
  namespace: capvrke2
spec:
  prefix: 16
  gateway: 172.16.0.1
  addresses:
    - 172.16.1.31-172.16.1.100
---
apiVersion: ipam.cluster.x-k8s.io/v1alpha2
kind: InClusterIPPool
metadata:
  name: capvrke2wb
  namespace: capvrke2
spec:
  prefix: 16
  gateway: 172.16.0.1
  addresses:
    - 172.16.1.101-172.16.1.170
---
apiVersion: v1
kind: Secret
metadata:
  name: cloud-provider-credentials-vesx01
  namespace: 'capvrke2'
stringData:
  data: |-
    apiVersion: v1
    kind: Secret
    metadata:
      labels:
        component: cloud-controller-manager
        vsphere-cpi-infra: secret
      name: cloud-provider-credentials-vesx01
      namespace: kube-system
    stringData:
      vesx01.domain.com.password: "***"
      vesx01.domain.com.username: "***"
    type: Opaque
type: addons.cluster.x-k8s.io/resource-set
---
apiVersion: v1
kind: Secret
metadata:
  name: cloud-provider-credentials-vesx02
  namespace: 'capvrke2'
stringData:
  data: |-
    apiVersion: v1
    kind: Secret
    metadata:
      labels:
        component: cloud-controller-manager
        vsphere-cpi-infra: secret
      name: cloud-provider-credentials-vesx02
      namespace: kube-system
    stringData:
      vesx02.domain.com.password: "***"
      vesx02.domain.com.username: "***"
    type: Opaque
type: addons.cluster.x-k8s.io/resource-set
---
apiVersion: v1
kind: Secret
metadata:
  name: 'capvrke2'
  namespace: 'capvrke2'
stringData:
  password: "***"
  username: "***"
---
apiVersion: v1
kind: Secret
metadata:
  name: etcd-backup-credentials
  namespace: 'capvrke2'
type: Opaque
data:
  aws_access_key_id: ***
  aws_secret_access_key: ***
---
apiVersion: v1
kind: Secret
metadata:
  name: 'docker-registry-credentials'
  namespace: 'capvrke2'
data:
  username: ***
  password: ***
---
apiVersion: v1
data:
  data: |-
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: cloud-controller-manager
      labels:
        app: vsphere-cpi
        vsphere-cpi-infra: service-account
        component: cloud-controller-manager
      namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: cloud-controller-manager
      labels:
        app: vsphere-cpi
        vsphere-cpi-infra: role
        component: cloud-controller-manager
    rules:
      - apiGroups:
          - ""
        resources:
          - events
        verbs:
          - create
          - patch
          - update
      - apiGroups:
          - ""
        resources:
          - nodes
        verbs:
          - "*"
      - apiGroups:
          - ""
        resources:
          - nodes/status
        verbs:
          - patch
      - apiGroups:
          - ""
        resources:
          - services
        verbs:
          - list
          - patch
          - update
          - watch
      - apiGroups:
          - ""
        resources:
          - services/status
        verbs:
          - patch
      - apiGroups:
          - ""
        resources:
          - serviceaccounts
        verbs:
          - create
          - get
          - list
          - watch
          - update
      - apiGroups:
          - ""
        resources:
          - persistentvolumes
        verbs:
          - get
          - list
          - update
          - watch
      - apiGroups:
          - ""
        resources:
          - endpoints
        verbs:
          - create
          - get
          - list
          - watch
          - update
      - apiGroups:
          - ""
        resources:
          - secrets
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - "coordination.k8s.io"
        resources:
          - leases
        verbs:
          - create
          - get
          - list
          - watch
          - update
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: vsphere-cpi
      labels:
        app: vsphere-cpi
        vsphere-cpi-infra: daemonset
        component: cloud-controller-manager
        tier: control-plane
      namespace: kube-system
      annotations:
    spec:
      selector:
        matchLabels:
          app: vsphere-cpi
      updateStrategy:
        type: RollingUpdate
      template:
        metadata:
          labels:
            app: vsphere-cpi
            component: cloud-controller-manager
            tier: control-plane
            release: release-name
            vsphere-cpi-infra: daemonset
        spec:
          tolerations:
            - effect: NoSchedule
              key: node.cloudprovider.kubernetes.io/uninitialized
              value: "true"
            - effect: NoSchedule
              key: node-role.kubernetes.io/control-plane
              operator: Exists
            - effect: NoSchedule
              key: node.kubernetes.io/not-ready
              operator: Exists
            - effect: NoExecute
              key: CriticalAddonsOnly
              operator: Exists
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: node-role.kubernetes.io/control-plane
                    operator: Exists
          securityContext:
            fsGroup: 1001
            runAsUser: 1001
          serviceAccountName: cloud-controller-manager
          hostNetwork: true
          dnsPolicy: ClusterFirst
          priorityClassName: system-node-critical
          containers:
          - name: vsphere-cpi
            image: registry.k8s.io/cloud-pv-vsphere/cloud-provider-vsphere:v1.32.2
            imagePullPolicy: IfNotPresent
            args:
              - --cloud-provider=vsphere
              - --v=2
              - --cloud-config=/etc/cloud/vsphere.conf
            volumeMounts:
              - mountPath: /etc/cloud
                name: vsphere-config-volume
                readOnly: true
          volumes:
            - name: vsphere-config-volume
              configMap:
                name: cloud-config
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      labels:
        app: vsphere-cpi
        component: cloud-controller-manager
        vsphere-cpi-infra: role-binding
      name: servicecatalog.k8s.io:apiserver-authentication-reader
      namespace: kube-system
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: extension-apiserver-authentication-reader
    subjects:
    - apiGroup: ""
      kind: ServiceAccount
      name: cloud-controller-manager
      namespace: kube-system
    - apiGroup: ""
      kind: User
      name: cloud-controller-manager
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      labels:
        app: vsphere-cpi
        component: cloud-controller-manager
        vsphere-cpi-infra: cluster-role-binding
      name: cloud-controller-manager
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cloud-controller-manager
    subjects:
    - kind: ServiceAccount
      name: cloud-controller-manager
      namespace: kube-system
    - kind: User
      name: cloud-controller-manager
    ---
    apiVersion: v1
    data:
      vsphere.conf: |
        global:
          port: 443
          insecureFlag: true
        vcenter:
          vesx01.domain.com:
            secretName: cloud-provider-credentials-vesx01
            secretNamespace: kube-system
            datacenters:
            - 'DC01'
            server: 'vesx01.domain.com'
          vesx02.domain.com:
            secretName: cloud-provider-credentials-vesx02
            secretNamespace: kube-system
            datacenters:
            - 'DC02'
            server: 'vesx02.domain.com'
        labels:
          region: k8s-region
          zone: k8s-zone
    kind: ConfigMap
    metadata:
      name: cloud-config
      namespace: kube-system
kind: ConfigMap
metadata:
  name: cpi-manifests
  namespace: 'capvrke2'
---
apiVersion: addons.cluster.x-k8s.io/v1beta1
kind: ClusterResourceSet
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: 'capvrke2'
  name: capvrke2-crs-0
  namespace: 'capvrke2'
spec:
  clusterSelector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: 'capvrke2'
  resources:
    - kind: Secret
      name: cloud-provider-credentials-vesx01
    - kind: Secret
      name: cloud-provider-credentials-vesx02
    - kind: ConfigMap
      name: cpi-manifests
  strategy: Reconcile
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: RKE2ConfigTemplate
metadata:
  name: 'capvrke2wa'
  namespace: 'capvrke2'
spec:
  template:
    spec:
      preRKE2Commands:
        - sleep 15
        - 'echo "[keyfile]" > /etc/NetworkManager/conf.d/calico.conf'
        - 'echo "unmanaged-devices=interface-name:cali*;interface-name:tunl*" >> /etc/NetworkManager/conf.d/calico.conf'
        - systemctl reload NetworkManager
      privateRegistriesConfig:
        configs:
          "registry-1.docker.io":
            authSecret:
              apiVersion: v1
              kind: Secret
              namespace: capvrke2
              name: docker-registry-credentials
      agentConfig:
        additionalUserData:
          config: |
            users:
              - name: capv
                sudo: ALL=(ALL) NOPASSWD:ALL
                ssh-authorized-keys:
                  - ssh-rsa ***
                  - ssh-rsa ***
                  - ssh-rsa ***
        kubelet:
          extraArgs:
            - --cloud-provider=external
        cisProfile: cis
        podSecurityAdmissionConfigFile: /etc/rancher/rke2/rke2-pss.yaml
      files:
        - path: /etc/rancher/rke2/rke2-pss.yaml
          owner: "root:root"
          permissions: "0640"
          content: |
            apiVersion: apiserver.config.k8s.io/v1
            kind: AdmissionConfiguration
            plugins:
            - name: PodSecurity
              configuration:
                apiVersion: pod-security.admission.config.k8s.io/v1beta1
                kind: PodSecurityConfiguration
                defaults:
                  enforce: "privileged"
                  enforce-version: "latest"
                  audit: "restricted"
                  audit-version: "latest"
                  warn: "restricted"
                  warn-version: "latest"
                exemptions:
                  usernames: []
                  runtimeClasses: []
                  namespaces: [calico-system, cattle-fleet-system, cattle-impersonation-system, cattle-system, cattle-ui-plugin-system, default, fleet-addon-agent, kube-node-lease, kube-public, kube-system, local, tigera-operator]
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: RKE2ConfigTemplate
metadata:
  name: 'capvrke2wb'
  namespace: 'capvrke2'
spec:
  template:
    spec:
      preRKE2Commands:
        - sleep 15
        - 'echo "[keyfile]" > /etc/NetworkManager/conf.d/calico.conf'
        - 'echo "unmanaged-devices=interface-name:cali*;interface-name:tunl*" >> /etc/NetworkManager/conf.d/calico.conf'
        - systemctl reload NetworkManager
      privateRegistriesConfig:
        configs:
          "registry-1.docker.io":
            authSecret:
              apiVersion: v1
              kind: Secret
              namespace: capvrke2
              name: docker-registry-credentials
      agentConfig:
        additionalUserData:
          config: |
            users:
              - name: capv
                sudo: ALL=(ALL) NOPASSWD:ALL
                ssh-authorized-keys:
                  - ssh-rsa ***
                  - ssh-rsa ***
                  - ssh-rsa ***
        kubelet:
          extraArgs:
            - --cloud-provider=external
        cisProfile: cis
        podSecurityAdmissionConfigFile: /etc/rancher/rke2/rke2-pss.yaml
      files:
        - path: /etc/rancher/rke2/rke2-pss.yaml
          owner: "root:root"
          permissions: "0640"
          content: |
            apiVersion: apiserver.config.k8s.io/v1
            kind: AdmissionConfiguration
            plugins:
            - name: PodSecurity
              configuration:
                apiVersion: pod-security.admission.config.k8s.io/v1beta1
                kind: PodSecurityConfiguration
                defaults:
                  enforce: "privileged"
                  enforce-version: "latest"
                  audit: "restricted"
                  audit-version: "latest"
                  warn: "restricted"
                  warn-version: "latest"
                exemptions:
                  usernames: []
                  runtimeClasses: []
                  namespaces: [calico-system, cattle-fleet-system, cattle-impersonation-system, cattle-system, cattle-ui-plugin-system, default, fleet-addon-agent, kube-node-lease, kube-public, kube-system, local, tigera-operator]
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  name: 'capvrke2m'
  namespace: 'capvrke2'
spec:
  template:
    spec:
      cloneMode: fullClone
      datacenter: 'DC01'
      datastore: 'DS01'
      diskGiB: 100
      folder: 'capvrke2'
      memoryMiB: 32768
      network:
        devices:
          - addressesFromPools:
              - apiGroup: ipam.cluster.x-k8s.io
                kind: InClusterIPPool
                name: 'capvrke2m'
            nameservers:
              - 172.16.1.20
              - 172.16.1.21
            networkName: 'VLAN_NET'
            searchDomains:
              - domain.com
            dhcp4: false
            dhcp6: false
      numCPUs: 8
      os: Linux
      powerOffMode: hard
      resourcePool: 'R01/Resources'
      server: 'vesx01.domain.com'
      storagePolicyName: ''
      template: 'sl-micro-6-1-base'
      thumbprint: '***'
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  name: 'capvrke2wa'
  namespace: 'capvrke2'
spec:
  template:
    spec:
      cloneMode: fullClone
      datacenter: 'DC01'
      datastore: 'DS01'
      diskGiB: 125
      folder: 'capvrke2'
      memoryMiB: 65536
      network:
        devices:
          - addressesFromPools:
              - apiGroup: ipam.cluster.x-k8s.io
                kind: InClusterIPPool
                name: 'capvrke2wa'
            nameservers:
              - 172.16.1.20
              - 172.16.1.21
            networkName: 'VLAN_NET'
            searchDomains:
              - domain.com
            dhcp4: false
            dhcp6: false
      numCPUs: 16
      os: Linux
      powerOffMode: hard
      resourcePool: 'R01/Resources'
      server: 'vesx01.domain.com'
      storagePolicyName: ''
      template: 'sl-micro-6-1-base'
      thumbprint: '***'
      failureDomain: fd01
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  name: 'capvrke2wb'
  namespace: 'capvrke2'
spec:
  template:
    spec:
      cloneMode: fullClone
      datacenter: 'DC02'
      datastore: 'DS02'
      diskGiB: 125
      folder: 'capvrke2'
      memoryMiB: 65536
      network:
        devices:
          - addressesFromPools:
              - apiGroup: ipam.cluster.x-k8s.io
                kind: InClusterIPPool
                name: 'capvrke2wb'
            nameservers:
              - 172.16.1.20
              - 172.16.1.21
            networkName: 'VLAN_NET'
            searchDomains:
              - domain.com
            dhcp4: false
            dhcp6: false
      numCPUs: 16
      os: Linux
      powerOffMode: hard
      resourcePool: 'R02/Resources'
      server: 'vesx02.domain.com'
      storagePolicyName: ''
      template: 'sl-micro-6-1-base'
      thumbprint: '***'
      failureDomain: fd02
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: RKE2ControlPlane
metadata:
  name: 'capvrke2m'
  namespace: 'capvrke2'
spec:
  serverConfig:
    tlsSan:
      - 'm.capvrke2.domain.com'
    cni: calico
    disableComponents:
      kubernetesComponents:
        - cloudController
      pluginComponents:
        - rke2-ingress-nginx
    etcd:
      exposeMetrics: true
      backupConfig:
        retention: "24"
        scheduleCron: "0 */3 * * *"
        s3:
          bucket: backups-rancher
          endpoint: cephbakedi05.domain.com
          enforceSslVerify: true
          s3CredentialSecret:
            apiVersion: v1
            kind: Secret
            namespace: capvrke2
            name: etcd-backup-credentials
  files:
    - path: "/var/lib/rancher/rke2/server/manifests/coredns-config.yaml"
      owner: "root:root"
      permissions: "0640"
      content: |
        apiVersion: helm.cattle.io/v1
        kind: HelmChartConfig
        metadata:
          name: rke2-coredns
          namespace: kube-system
        spec:
          valuesContent: |-
            tolerations:
              - key: "node.cloudprovider.kubernetes.io/uninitialized"
                value: "true"
                effect: "NoSchedule"
            autoscaler:
              enable: true
              min: 4
    - path: "/var/lib/rancher/rke2/server/manifests/kubevip.yaml"
      owner: "root:root"
      permissions: "0640"
      content: |
        apiVersion: v1
        kind: ServiceAccount
        metadata:
          name: kube-vip
          namespace: kube-system
        ---
        apiVersion: rbac.authorization.k8s.io/v1
        kind: ClusterRole
        metadata:
          annotations:
            rbac.authorization.kubernetes.io/autoupdate: "true"
          name: system:kube-vip-role
        rules:
          - apiGroups: [""]
            resources: ["services/status"]
            verbs: ["update"]
          - apiGroups: [""]
            resources: ["services", "endpoints"]
            verbs: ["list","get","watch", "update"]
          - apiGroups: [""]
            resources: ["nodes"]
            verbs: ["list","get","watch", "update", "patch"]
          - apiGroups: ["coordination.k8s.io"]
            resources: ["leases"]
            verbs: ["list", "get", "watch", "update", "create"]
          - apiGroups: ["discovery.k8s.io"]
            resources: ["endpointslices"]
            verbs: ["list","get","watch", "update"]
        ---
        kind: ClusterRoleBinding
        apiVersion: rbac.authorization.k8s.io/v1
        metadata:
          name: system:kube-vip-binding
        roleRef:
          apiGroup: rbac.authorization.k8s.io
          kind: ClusterRole
          name: system:kube-vip-role
        subjects:
        - kind: ServiceAccount
          name: kube-vip
          namespace: kube-system
        ---
        apiVersion: apps/v1
        kind: DaemonSet
        metadata:
          creationTimestamp: null
          labels:
            app.kubernetes.io/name: kube-vip-ds
            app.kubernetes.io/version: v0.8.7
          name: kube-vip-ds
          namespace: kube-system
        spec:
          selector:
            matchLabels:
              app.kubernetes.io/name: kube-vip-ds
          template:
            metadata:
              creationTimestamp: null
              labels:
                app.kubernetes.io/name: kube-vip-ds
                app.kubernetes.io/version: v0.8.7
            spec:
              affinity:
                nodeAffinity:
                  requiredDuringSchedulingIgnoredDuringExecution:
                    nodeSelectorTerms:
                    - matchExpressions:
                      - key: node-role.kubernetes.io/master
                        operator: Exists
                    - matchExpressions:
                      - key: node-role.kubernetes.io/control-plane
                        operator: Exists
              containers:
              - args:
                - manager
                env:
                - name: vip_arp
                  value: "true"
                - name: port
                  value: "6443"
                - name: vip_nodename
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
                - name: vip_interface
                  value: "eth0"
                - name: vip_servicesinterface
                  value: "eth0"
                - name: vip_cidr
                  value: "32"
                - name: dns_mode
                  value: first
                - name: cp_enable
                  value: "true"
                - name: cp_namespace
                  value: kube-system
                - name: svc_enable
                  value: "true"
                - name: svc_leasename
                  value: plndr-svcs-lock
                - name: vip_leaderelection
                  value: "true"
                - name: vip_leasename
                  value: plndr-cp-lock
                - name: vip_leaseduration
                  value: "5"
                - name: vip_renewdeadline
                  value: "3"
                - name: vip_retryperiod
                  value: "1"
                - name: address
                  value: 172.16.1.254
                image: ghcr.io/kube-vip/kube-vip:v0.8.7
                imagePullPolicy: IfNotPresent
                name: kube-vip
                resources: {}
                securityContext:
                  capabilities:
                    add:
                    - NET_ADMIN
                    - NET_RAW
              hostNetwork: true
              serviceAccountName: kube-vip
              tolerations:
              - effect: NoSchedule
                operator: Exists
              - effect: NoExecute
                operator: Exists
          updateStrategy: {}
    - path: /etc/rancher/rke2/rke2-pss.yaml
      owner: "root:root"
      permissions: "0640"
      content: |
        apiVersion: apiserver.config.k8s.io/v1
        kind: AdmissionConfiguration
        plugins:
        - name: PodSecurity
          configuration:
            apiVersion: pod-security.admission.config.k8s.io/v1beta1
            kind: PodSecurityConfiguration
            defaults:
              enforce: "privileged"
              enforce-version: "latest"
              audit: "restricted"
              audit-version: "latest"
              warn: "restricted"
              warn-version: "latest"
            exemptions:
              usernames: []
              runtimeClasses: []
              namespaces: [calico-system, cattle-fleet-system, cattle-impersonation-system, cattle-system, cattle-ui-plugin-system, default, fleet-addon-agent, kube-node-lease, kube-public, kube-system, local, tigera-operator]
    - path: "/var/lib/rancher/rke2/server/manifests/rke2-calico-config.yaml"
      owner: "root:root"
      permissions: "0640"
      content: |
        apiVersion: helm.cattle.io/v1
        kind: HelmChartConfig
        metadata:
          name: rke2-calico
          namespace: kube-system
        spec:
          valuesContent: |-
            installation:
              calicoNetwork:
                bgp: Enabled
                ipPools:
                  - cidr: 10.80.0.0/16
                    encapsulation: IPIP
                    natOutgoing: Enabled
            felixConfiguration:
              featureDetectOverride: "ChecksumOffloadBroken=false"
  replicas: 5
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: VSphereMachineTemplate
    name: 'capvrke2m'
  nodeDrainTimeout: 30s
  preRKE2Commands:
    - sleep 15
    - 'echo "[keyfile]" > /etc/NetworkManager/conf.d/calico.conf'
    - 'echo "unmanaged-devices=interface-name:cali*;interface-name:tunl*" >> /etc/NetworkManager/conf.d/calico.conf'
    - systemctl reload NetworkManager
  privateRegistriesConfig:
    configs:
      "registry-1.docker.io":
        authSecret:
          apiVersion: v1
          kind: Secret
          namespace: capvrke2
          name: docker-registry-credentials
  version: 'v1.32.5+rke2r1'
  agentConfig:
    additionalUserData:
      config: |
        users:
          - name: capv
            sudo: ALL=(ALL) NOPASSWD:ALL
            ssh-authorized-keys:
              - ssh-rsa ***
              - ssh-rsa ***
              - ssh-rsa ***
    kubelet:
      extraArgs:
        - --cloud-provider=external
    cisProfile: cis
    podSecurityAdmissionConfigFile: /etc/rancher/rke2/rke2-pss.yaml
  registrationMethod: "control-plane-endpoint"
  rolloutStrategy:
    type: "RollingUpdate"
    rollingUpdate:
      maxSurge: 1
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: 'capvrke2'
  name: 'capvrke2wa'
  namespace: 'capvrke2'
spec:
  clusterName: 'capvrke2'
  replicas: 3
  selector:
    matchLabels: {}
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: 'capvrke2'
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: RKE2ConfigTemplate
          name: 'capvrke2wa'
      clusterName: 'capvrke2'
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: VSphereMachineTemplate
        name: capvrke2wa
      nodeDeletionTimeout: 10s
      nodeDrainTimeout: 1m0s
      nodeVolumeDetachTimeout: 1m0s
      version: 'v1.32.5+rke2r1'
  strategy:
    rollingUpdate:
      deletePolicy: "Oldest"
      maxSurge: 1
      maxUnavailable: 1
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: 'capvrke2'
  name: 'capvrke2wb'
  namespace: 'capvrke2'
spec:
  clusterName: 'capvrke2'
  replicas: 3
  selector:
    matchLabels: {}
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: 'capvrke2'
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: RKE2ConfigTemplate
          name: 'capvrke2wb'
      clusterName: 'capvrke2'
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: VSphereMachineTemplate
        name: capvrke2wb
      nodeDeletionTimeout: 10s
      nodeDrainTimeout: 1m0s
      nodeVolumeDetachTimeout: 1m0s
      version: 'v1.32.5+rke2r1'
  strategy:
    rollingUpdate:
      deletePolicy: "Oldest"
      maxSurge: 1
      maxUnavailable: 1
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereCluster
metadata:
  name: 'capvrke2'
  namespace: 'capvrke2'
spec:
  controlPlaneEndpoint:
    host: 172.16.1.254
    port: 6443
  identityRef:
    kind: Secret
    name: 'capvrke2'
  server: 'vesx01.domain.com'
  thumbprint: '***'
  failureDomainSelector: {}
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: 'capvrke2'
  name: 'capvrke2'
  namespace: 'capvrke2'
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - 10.80.0.0/16
    services:
      cidrBlocks:
        - 10.90.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: RKE2ControlPlane
    name: 'capvrke2m'
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: VSphereCluster
    name: 'capvrke2'
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereDeploymentZone
metadata:
  name: fd01
spec:
  controlPlane: true
  failureDomain: fd01
  placementConstraint:
    resourcePool: R01/Resources
  server: vesx01.domain.com
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereDeploymentZone
metadata:
  name: fd02
spec:
  controlPlane: true
  failureDomain: fd02
  placementConstraint:
    resourcePool: R02/Resources
  server: vesx02.domain.com
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereFailureDomain
metadata:
  name: fd01
spec:
  region:
    autoConfigure: false
    name: dc01
    tagCategory: k8s-region
    type: Datacenter
  topology:
    computeCluster: R01
    datacenter: DC01
    datastore: DS01
    networks:
    - VLAN_NET
  zone:
    autoConfigure: false
    name: fd01
    tagCategory: k8s-zone
    type: ComputeCluster
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereFailureDomain
metadata:
  name: fd02
spec:
  region:
    autoConfigure: false
    name: dc02
    tagCategory: k8s-region
    type: Datacenter
  topology:
    computeCluster: R02
    datacenter: DC02
    datastore: DS02
    networks:
    - VLAN_NET
  zone:
    autoConfigure: false
    name: fd02
    tagCategory: k8s-zone
    type: ComputeCluster

Regards.

[chrischdi]

Can you also provide the resulting Machine and VSphereMachine objects?

I don't have a vsphere env on hand so can't try to reproduce it.

[jisnardo]

Hi @chrischdi, here it is.

# k get machine

NAME                     CLUSTER    NODENAME                 PROVIDERID                                       PHASE     AGE   VERSION
capvrke2m-58th8          capvrke2   capvrke2m-58th8          vsphere://4234f376-a76a-b1ba-b374-0c37a1413b70   Running   16h   v1.32.5+rke2r1 - vesx01.domain.com
capvrke2m-86ncg          capvrke2   capvrke2m-86ncg          vsphere://4234227d-40c8-f32a-180f-7cf0504ca8cc   Running   16h   v1.32.5+rke2r1 - vesx01.domain.com
capvrke2m-kp9r2          capvrke2   capvrke2m-kp9r2          vsphere://4234a844-724a-1fa9-2544-e095e613d3cb   Running   16h   v1.32.5+rke2r1 - vesx01.domain.com
capvrke2m-prnxs          capvrke2   capvrke2m-prnxs          vsphere://4234709e-f3db-98e3-3954-0f6dd7a4b08f   Running   16h   v1.32.5+rke2r1 - vesx01.domain.com
capvrke2m-vzrqr          capvrke2   capvrke2m-vzrqr          vsphere://42340ffd-05ab-df59-79bb-981275997a1e   Running   16h   v1.32.5+rke2r1 - vesx01.domain.com
capvrke2wa-mclld-98z8x   capvrke2   capvrke2wa-mclld-98z8x   vsphere://4234529b-d0a0-4859-e157-613d38dbc021   Running   16h   v1.32.5+rke2r1 - vesx01.domain.com
capvrke2wa-mclld-l2nks   capvrke2   capvrke2wa-mclld-l2nks   vsphere://4234ab52-352f-166b-1fd0-bcb5d6aac11b   Running   16h   v1.32.5+rke2r1 - vesx01.domain.com
capvrke2wa-mclld-mw54q   capvrke2   capvrke2wa-mclld-mw54q   vsphere://423489bb-2862-4b45-6ced-95e65690e349   Running   16h   v1.32.5+rke2r1 - vesx01.domain.com
capvrke2wb-sg8z7-kvhzx   capvrke2   capvrke2wb-sg8z7-kvhzx   vsphere://4228f970-670e-2641-bae8-a0ded44b7ab0   Running   16h   v1.32.5+rke2r1 - vesx02.domain.com

# k get machine capvrke2m-58th8 -o yaml

apiVersion: cluster.x-k8s.io/v1beta1
kind: Machine
metadata:
  creationTimestamp: "2025-09-08T14:45:27Z"
  finalizers:
  - machine.cluster.x-k8s.io
  generation: 3
  labels:
    cluster.x-k8s.io/cluster-name: capvrke2
    cluster.x-k8s.io/control-plane: ""
    cluster.x-k8s.io/control-plane-name: capvrke2m
    node.cluster.x-k8s.io/esxi-host: 172.16.24.9
  name: capvrke2m-58th8
  namespace: capvrke2
  ownerReferences:
  - apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: RKE2ControlPlane
    name: capvrke2m
    uid: 778731d9-95a3-4b67-955d-e8302d816d15
  resourceVersion: "409651209"
  uid: f61237cb-27ab-4e0e-84d6-bf195bab7c4c
spec:
  bootstrap:
    configRef:
      apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
      kind: RKE2Config
      name: capvrke2m-8v7kc
      namespace: capvrke2
      uid: 972663a9-c719-4f1d-bf99-309a3ffe5b11
    dataSecretName: capvrke2m-8v7kc
  clusterName: capvrke2
  failureDomain: fd01
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: VSphereMachine
    name: capvrke2m-266sh
    namespace: capvrke2
    uid: 05f1d498-1865-4f52-83f4-4c72e54f7357
  nodeDeletionTimeout: 10s
  nodeDrainTimeout: 30s
  nodeVolumeDetachTimeout: 5m0s
  providerID: vsphere://4234f376-a76a-b1ba-b374-0c37a1413b70
  version: v1.32.5+rke2r1
status:
  addresses:
  - address: 172.16.1.5
    type: ExternalIP
  - address: capvrke2m-58th8
    type: InternalDNS
  bootstrapReady: true
  conditions:
  - lastTransitionTime: "2025-09-08T14:46:36Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-09-08T14:46:56Z"
    status: "True"
    type: AgentHealthy
  - lastTransitionTime: "2025-09-08T14:45:27Z"
    status: "True"
    type: BootstrapReady
  - lastTransitionTime: "2025-09-08T14:46:56Z"
    status: "True"
    type: EtcdMemberHealthy
  - lastTransitionTime: "2025-09-08T14:46:36Z"
    status: "True"
    type: InfrastructureReady
  - lastTransitionTime: "2025-09-08T14:46:54Z"
    status: "True"
    type: NodeHealthy
  - lastTransitionTime: "2025-09-08T14:46:56Z"
    status: "True"
    type: NodeMetadataUpToDate
  infrastructureReady: true
  lastUpdated: "2025-09-08T14:46:39Z"
  nodeInfo:
    architecture: amd64
    bootID: 4bdec20d-a0e4-436c-a89d-b1065b7cd1d1
    containerRuntimeVersion: containerd://2.0.5-k3s1
    kernelVersion: 6.4.0-19-default
    kubeProxyVersion: v1.32.5+rke2r1
    kubeletVersion: v1.32.5+rke2r1
    machineID: 8dd656d055964e12ab61b861a0158b05
    operatingSystem: linux
    osImage: SUSE Linux Micro 6.1
    systemUUID: 76f33442-6aa7-bab1-b374-0c37a1413b70
  nodeRef:
    apiVersion: v1
    kind: Node
    name: capvrke2m-58th8
    uid: ae5e23cc-75a2-41b8-bbff-ce5697a3a6f6
  observedGeneration: 3
  phase: Running
  v1beta2:
    conditions:
    - lastTransitionTime: "2025-09-08T14:46:54Z"
      message: ""
      observedGeneration: 3
      reason: Available
      status: "True"
      type: Available
    - lastTransitionTime: "2025-09-08T14:46:54Z"
      message: ""
      observedGeneration: 3
      reason: Ready
      status: "True"
      type: Ready
    - lastTransitionTime: "2025-09-08T14:45:27Z"
      message: ""
      observedGeneration: 3
      reason: Ready
      status: "True"
      type: BootstrapConfigReady
    - lastTransitionTime: "2025-09-08T14:46:35Z"
      message: ""
      observedGeneration: 3
      reason: Ready
      status: "True"
      type: InfrastructureReady
    - lastTransitionTime: "2025-09-08T14:46:54Z"
      message: ""
      observedGeneration: 3
      reason: NodeHealthy
      status: "True"
      type: NodeHealthy
    - lastTransitionTime: "2025-09-08T14:46:54Z"
      message: ""
      observedGeneration: 3
      reason: NodeReady
      status: "True"
      type: NodeReady
    - lastTransitionTime: "2025-09-08T14:45:27Z"
      message: ""
      observedGeneration: 3
      reason: NotPaused
      status: "False"
      type: Paused
    - lastTransitionTime: "2025-09-08T14:45:27Z"
      message: ""
      observedGeneration: 3
      reason: NotDeleting
      status: "False"
      type: Deleting

# k get machine capvrke2wa-mclld-98z8x -o yaml

apiVersion: cluster.x-k8s.io/v1beta1
kind: Machine
metadata:
  creationTimestamp: "2025-09-08T14:38:28Z"
  finalizers:
  - machine.cluster.x-k8s.io
  generation: 3
  labels:
    cluster.x-k8s.io/cluster-name: capvrke2
    cluster.x-k8s.io/deployment-name: capvrke2wa
    cluster.x-k8s.io/set-name: capvrke2wa-mclld
    machine-template-hash: 617449236-mclld
    node.cluster.x-k8s.io/esxi-host: 172.16.24.6
  name: capvrke2wa-mclld-98z8x
  namespace: capvrke2
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: MachineSet
    name: capvrke2wa-mclld
    uid: b4c5f7ae-a52b-4e8e-a4d0-8e217b42d432
  resourceVersion: "409644419"
  uid: db81f7df-62c3-4a6f-81e2-f9576e8133e0
spec:
  bootstrap:
    configRef:
      apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
      kind: RKE2Config
      name: capvrke2wa-mclld-98z8x
      namespace: capvrke2
      uid: 98c2a5dc-3c55-438d-8ce3-1b9d0c638d7e
    dataSecretName: capvrke2wa-mclld-98z8x
  clusterName: capvrke2
  failureDomain: fd01
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: VSphereMachine
    name: capvrke2wa-mclld-98z8x
    namespace: capvrke2
    uid: d02285d2-757a-42e3-8b8d-aba80e0030a3
  nodeDeletionTimeout: 10s
  nodeDrainTimeout: 1m0s
  nodeVolumeDetachTimeout: 1m0s
  providerID: vsphere://4234529b-d0a0-4859-e157-613d38dbc021
  version: v1.32.5+rke2r1
status:
  addresses:
  - address: 172.16.1.31
    type: ExternalIP
  - address: capvrke2wa-mclld-98z8x
    type: InternalDNS
  bootstrapReady: true
  conditions:
  - lastTransitionTime: "2025-09-08T14:39:39Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-09-08T14:38:29Z"
    status: "True"
    type: BootstrapReady
  - lastTransitionTime: "2025-09-08T14:39:39Z"
    status: "True"
    type: InfrastructureReady
  - lastTransitionTime: "2025-09-08T14:39:46Z"
    status: "True"
    type: NodeHealthy
  infrastructureReady: true
  lastUpdated: "2025-09-08T14:39:39Z"
  nodeInfo:
    architecture: amd64
    bootID: 5192f436-5e97-4b90-b6da-bf403a56f95e
    containerRuntimeVersion: containerd://2.0.5-k3s1
    kernelVersion: 6.4.0-19-default
    kubeProxyVersion: v1.32.5+rke2r1
    kubeletVersion: v1.32.5+rke2r1
    machineID: 8dd656d055964e12ab61b861a0158b05
    operatingSystem: linux
    osImage: SUSE Linux Micro 6.1
    systemUUID: 9b523442-a0d0-5948-e157-613d38dbc021
  nodeRef:
    apiVersion: v1
    kind: Node
    name: capvrke2wa-mclld-98z8x
    uid: 7a37299f-deee-47d4-a4e9-e00fe303f9ef
  observedGeneration: 3
  phase: Running
  v1beta2:
    conditions:
    - lastTransitionTime: "2025-09-08T14:39:46Z"
      message: ""
      observedGeneration: 3
      reason: Available
      status: "True"
      type: Available
    - lastTransitionTime: "2025-09-08T14:39:46Z"
      message: ""
      observedGeneration: 3
      reason: Ready
      status: "True"
      type: Ready
    - lastTransitionTime: "2025-09-08T14:38:32Z"
      message: ""
      observedGeneration: 3
      reason: UpToDate
      status: "True"
      type: UpToDate
    - lastTransitionTime: "2025-09-08T14:38:29Z"
      message: ""
      observedGeneration: 3
      reason: Ready
      status: "True"
      type: BootstrapConfigReady
    - lastTransitionTime: "2025-09-08T14:39:39Z"
      message: ""
      observedGeneration: 3
      reason: Ready
      status: "True"
      type: InfrastructureReady
    - lastTransitionTime: "2025-09-08T14:39:46Z"
      message: ""
      observedGeneration: 3
      reason: NodeHealthy
      status: "True"
      type: NodeHealthy
    - lastTransitionTime: "2025-09-08T14:39:46Z"
      message: ""
      observedGeneration: 3
      reason: NodeReady
      status: "True"
      type: NodeReady
    - lastTransitionTime: "2025-09-08T14:38:28Z"
      message: ""
      observedGeneration: 3
      reason: NotPaused
      status: "False"
      type: Paused
    - lastTransitionTime: "2025-09-08T14:38:28Z"
      message: ""
      observedGeneration: 3
      reason: NotDeleting
      status: "False"
      type: Deleting

# k get machine capvrke2wb-sg8z7-kvhzx -o yaml

apiVersion: cluster.x-k8s.io/v1beta1
kind: Machine
metadata:
  creationTimestamp: "2025-09-08T14:38:28Z"
  finalizers:
  - machine.cluster.x-k8s.io
  generation: 3
  labels:
    cluster.x-k8s.io/cluster-name: capvrke2
    cluster.x-k8s.io/deployment-name: capvrke2wb
    cluster.x-k8s.io/set-name: capvrke2wb-sg8z7
    machine-template-hash: 1684660654-sg8z7
    node.cluster.x-k8s.io/esxi-host: 172.16.21.3
  name: capvrke2wb-sg8z7-kvhzx
  namespace: capvrke2
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: MachineSet
    name: capvrke2wb-sg8z7
    uid: 238b0760-bab8-409b-bd5a-b6ab04ad016a
  resourceVersion: "409646127"
  uid: 3b0a7123-3963-490d-907d-ae2bd40bd047
spec:
  bootstrap:
    configRef:
      apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
      kind: RKE2Config
      name: capvrke2wb-sg8z7-kvhzx
      namespace: capvrke2
      uid: eb588623-6a4f-4073-81da-aba188a78785
    dataSecretName: capvrke2wb-sg8z7-kvhzx
  clusterName: capvrke2
  failureDomain: fd02
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: VSphereMachine
    name: capvrke2wb-sg8z7-kvhzx
    namespace: capvrke2
    uid: 490251fc-402b-4f51-83e0-12c6040f9e3f
  nodeDeletionTimeout: 10s
  nodeDrainTimeout: 1m0s
  nodeVolumeDetachTimeout: 1m0s
  providerID: vsphere://4228f970-670e-2641-bae8-a0ded44b7ab0
  version: v1.32.5+rke2r1
status:
  addresses:
  - address: 172.16.1.102
    type: ExternalIP
  - address: capvrke2wb-sg8z7-kvhzx
    type: InternalDNS
  bootstrapReady: true
  conditions:
  - lastTransitionTime: "2025-09-08T14:41:23Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-09-08T14:38:28Z"
    status: "True"
    type: BootstrapReady
  - lastTransitionTime: "2025-09-08T14:41:23Z"
    status: "True"
    type: InfrastructureReady
  - lastTransitionTime: "2025-09-08T14:41:28Z"
    status: "True"
    type: NodeHealthy
  infrastructureReady: true
  lastUpdated: "2025-09-08T14:41:26Z"
  nodeInfo:
    architecture: amd64
    bootID: 487e2ad9-ae11-4c48-b616-c21aaac13f46
    containerRuntimeVersion: containerd://2.0.5-k3s1
    kernelVersion: 6.4.0-19-default
    kubeProxyVersion: v1.32.5+rke2r1
    kubeletVersion: v1.32.5+rke2r1
    machineID: 8dd656d055964e12ab61b861a0158b05
    operatingSystem: linux
    osImage: SUSE Linux Micro 6.1
    systemUUID: 70f92842-0e67-4126-bae8-a0ded44b7ab0
  nodeRef:
    apiVersion: v1
    kind: Node
    name: capvrke2wb-sg8z7-kvhzx
    uid: 24738c5d-f6bf-4bd3-9cae-c3984955256f
  observedGeneration: 3
  phase: Running
  v1beta2:
    conditions:
    - lastTransitionTime: "2025-09-08T14:41:28Z"
      message: ""
      observedGeneration: 3
      reason: Available
      status: "True"
      type: Available
    - lastTransitionTime: "2025-09-08T14:41:28Z"
      message: ""
      observedGeneration: 3
      reason: Ready
      status: "True"
      type: Ready
    - lastTransitionTime: "2025-09-08T14:38:29Z"
      message: ""
      observedGeneration: 3
      reason: UpToDate
      status: "True"
      type: UpToDate
    - lastTransitionTime: "2025-09-08T14:38:28Z"
      message: ""
      observedGeneration: 3
      reason: Ready
      status: "True"
      type: BootstrapConfigReady
    - lastTransitionTime: "2025-09-08T14:41:23Z"
      message: ""
      observedGeneration: 3
      reason: Ready
      status: "True"
      type: InfrastructureReady
    - lastTransitionTime: "2025-09-08T14:41:28Z"
      message: ""
      observedGeneration: 3
      reason: NodeHealthy
      status: "True"
      type: NodeHealthy
    - lastTransitionTime: "2025-09-08T14:41:28Z"
      message: ""
      observedGeneration: 3
      reason: NodeReady
      status: "True"
      type: NodeReady
    - lastTransitionTime: "2025-09-08T14:38:28Z"
      message: ""
      observedGeneration: 3
      reason: NotPaused
      status: "False"
      type: Paused
    - lastTransitionTime: "2025-09-08T14:38:28Z"
      message: ""
      observedGeneration: 3
      reason: NotDeleting
      status: "False"
      type: Deleting

# k get VSphereMachine

NAME                     CLUSTER    READY   PROVIDERID                                       AGE
capvrke2m-266sh          capvrke2   true    vsphere://4234f376-a76a-b1ba-b374-0c37a1413b70   16h
capvrke2m-6rbbt          capvrke2   true    vsphere://42340ffd-05ab-df59-79bb-981275997a1e   16h
capvrke2m-d2pd2          capvrke2   true    vsphere://4234a844-724a-1fa9-2544-e095e613d3cb   16h
capvrke2m-kjmzp          capvrke2   true    vsphere://4234709e-f3db-98e3-3954-0f6dd7a4b08f   16h
capvrke2m-nkq2r          capvrke2   true    vsphere://4234227d-40c8-f32a-180f-7cf0504ca8cc   16h
capvrke2wa-mclld-98z8x   capvrke2   true    vsphere://4234529b-d0a0-4859-e157-613d38dbc021   16h
capvrke2wa-mclld-l2nks   capvrke2   true    vsphere://4234ab52-352f-166b-1fd0-bcb5d6aac11b   16h
capvrke2wa-mclld-mw54q   capvrke2   true    vsphere://423489bb-2862-4b45-6ced-95e65690e349   16h
capvrke2wb-sg8z7-kvhzx   capvrke2   true    vsphere://4228f970-670e-2641-bae8-a0ded44b7ab0   16h

# k get VSphereMachine capvrke2m-266sh -o yaml

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachine
metadata:
  annotations:
    cluster.x-k8s.io/cloned-from-groupkind: VSphereMachineTemplate.infrastructure.cluster.x-k8s.io
    cluster.x-k8s.io/cloned-from-name: capvrke2m
  creationTimestamp: "2025-09-08T14:45:27Z"
  finalizers:
  - vspheremachine.infrastructure.cluster.x-k8s.io
  generation: 2
  labels:
    cluster.x-k8s.io/cluster-name: capvrke2
    cluster.x-k8s.io/control-plane: ""
    cluster.x-k8s.io/control-plane-name: capvrke2m
  name: capvrke2m-266sh
  namespace: capvrke2
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Machine
    name: capvrke2m-58th8
    uid: f61237cb-27ab-4e0e-84d6-bf195bab7c4c
  resourceVersion: "409650868"
  uid: 05f1d498-1865-4f52-83f4-4c72e54f7357
spec:
  cloneMode: fullClone
  datacenter: DC01
  datastore: DS01
  diskGiB: 100
  folder: capvrke2
  memoryMiB: 32768
  network:
    devices:
    - addressesFromPools:
      - apiGroup: ipam.cluster.x-k8s.io
        kind: InClusterIPPool
        name: capvrke2m
      nameservers:
      - 172.16.1.20
      - 172.16.1.21
      networkName: VLAN_NET
      searchDomains:
      - domain.com
  numCPUs: 8
  os: Linux
  powerOffMode: hard
  providerID: vsphere://4234f376-a76a-b1ba-b374-0c37a1413b70
  resourcePool: R01/Resources
  server: vesx01.domain.com
  template: sl-micro-6-1-base
  thumbprint: ***
status:
  addresses:
  - address: 172.16.1.5
    type: ExternalIP
  - address: capvrke2m-58th8
    type: InternalDNS
  conditions:
  - lastTransitionTime: "2025-09-08T14:46:35Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-09-08T14:46:35Z"
    status: "True"
    type: VMProvisioned
  network:
  - connected: true
    ipAddrs:
    - 172.16.1.5
    macAddr: ***
    networkName: VLAN_NET
  ready: true

# k get VSphereMachine capvrke2wa-mclld-98z8x -o yamlç

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachine
metadata:
  annotations:
    cluster.x-k8s.io/cloned-from-groupkind: VSphereMachineTemplate.infrastructure.cluster.x-k8s.io
    cluster.x-k8s.io/cloned-from-name: capvrke2wa
  creationTimestamp: "2025-09-08T14:38:28Z"
  finalizers:
  - vspheremachine.infrastructure.cluster.x-k8s.io
  generation: 2
  labels:
    cluster.x-k8s.io/cluster-name: capvrke2
    cluster.x-k8s.io/deployment-name: capvrke2wa
    cluster.x-k8s.io/set-name: capvrke2wa-mclld
    machine-template-hash: 617449236-mclld
  name: capvrke2wa-mclld-98z8x
  namespace: capvrke2
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Machine
    name: capvrke2wa-mclld-98z8x
    uid: db81f7df-62c3-4a6f-81e2-f9576e8133e0
  resourceVersion: "409644207"
  uid: d02285d2-757a-42e3-8b8d-aba80e0030a3
spec:
  cloneMode: fullClone
  datacenter: DC01
  datastore: DS01
  diskGiB: 125
  failureDomain: fd01
  folder: capvrke2
  memoryMiB: 65536
  network:
    devices:
    - addressesFromPools:
      - apiGroup: ipam.cluster.x-k8s.io
        kind: InClusterIPPool
        name: capvrke2wa
      nameservers:
      - 172.16.1.20
      - 172.16.1.21
      networkName: VLAN_NET
      searchDomains:
      - domain.com
  numCPUs: 16
  os: Linux
  powerOffMode: hard
  providerID: vsphere://4234529b-d0a0-4859-e157-613d38dbc021
  resourcePool: R01/Resources
  server: vesx01.domain.com
  template: sl-micro-6-1-base
  thumbprint: ***
status:
  addresses:
  - address: 172.16.1.31
    type: ExternalIP
  - address: capvrke2wa-mclld-98z8x
    type: InternalDNS
  conditions:
  - lastTransitionTime: "2025-09-08T14:39:39Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-09-08T14:39:39Z"
    status: "True"
    type: VMProvisioned
  network:
  - connected: true
    ipAddrs:
    - 172.16.1.31
    macAddr: ***
    networkName: VLAN_NET
  ready: true

# k get VSphereMachine capvrke2wb-sg8z7-kvhzx -o yaml

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachine
metadata:
  annotations:
    cluster.x-k8s.io/cloned-from-groupkind: VSphereMachineTemplate.infrastructure.cluster.x-k8s.io
    cluster.x-k8s.io/cloned-from-name: capvrke2wb
  creationTimestamp: "2025-09-08T14:38:28Z"
  finalizers:
  - vspheremachine.infrastructure.cluster.x-k8s.io
  generation: 2
  labels:
    cluster.x-k8s.io/cluster-name: capvrke2
    cluster.x-k8s.io/deployment-name: capvrke2wb
    cluster.x-k8s.io/set-name: capvrke2wb-sg8z7
    machine-template-hash: 1684660654-sg8z7
  name: capvrke2wb-sg8z7-kvhzx
  namespace: capvrke2
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Machine
    name: capvrke2wb-sg8z7-kvhzx
    uid: 3b0a7123-3963-490d-907d-ae2bd40bd047
  resourceVersion: "409645990"
  uid: 490251fc-402b-4f51-83e0-12c6040f9e3f
spec:
  cloneMode: fullClone
  datacenter: DC02
  datastore: DS02
  diskGiB: 125
  failureDomain: fd02
  folder: capvrke2
  memoryMiB: 65536
  network:
    devices:
    - addressesFromPools:
      - apiGroup: ipam.cluster.x-k8s.io
        kind: InClusterIPPool
        name: capvrke2wb
      nameservers:
      - 172.16.1.20
      - 172.16.1.21
      networkName: VLAN_NET
      searchDomains:
      - domain.com
  numCPUs: 16
  os: Linux
  powerOffMode: hard
  providerID: vsphere://4228f970-670e-2641-bae8-a0ded44b7ab0
  resourcePool: R02/Resources
  server: vesx02.domain.com
  template: sl-micro-6-1-base
  thumbprint: ***
status:
  addresses:
  - address: 172.16.1.102
    type: ExternalIP
  - address: capvrke2wb-sg8z7-kvhzx
    type: InternalDNS
  conditions:
  - lastTransitionTime: "2025-09-08T14:41:23Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-09-08T14:41:23Z"
    status: "True"
    type: VMProvisioned
  network:
  - connected: true
    ipAddrs:
    - 172.16.1.102
    macAddr: ***
    networkName: VLAN_NET
  ready: true

[chrischdi]

Looks for me like this is not an issue of CAPV, but an issue of the RKE2 ControlPlane provider.

The CAPI contract states the following:

Control Plane providers are expected to place machines in failure domains defined in Cluster’s status.failureDomains field.

More specifically, Control Plane should be spread across failure domains specifically flagged to host control plane machines.

Source: https://main.cluster-api.sigs.k8s.io/developer/providers/contracts/control-plane#machine-placement

I currently assume (because I don't have the information of all control-plane machines) that RKE2 does not set different failuredomains for your control plane machine objcets.

Happy to verify if you provide the full output of kubectl get machines,vspheremachines -o yaml

[jisnardo]

Maybe the question could be like this:

Q: Is there a way I can tell CAPV to distribute my control plane across multiple VSphereDeploymentZones?

A:

1. All the VSphereDpeloymentZones having the spec.Server field same as the spec.Server field for the VSphereCluster object will be used to distribute the CP nodes. This association happens automatically when the VSphereCluster and DeploymentZone objects are reconciled.

2. So the CP nodes will get deployed into the multiple failure domains automatically by the KCP which takes its value from the Cluster object which takes them from the VSphereCluster object - And IIRC, KCP does a round robin on the associated failure domains.

[chrischdi]

Maybe the question could be like this:

Q: Is there a way I can tell CAPV to distribute my control plane across multiple VSphereDeploymentZones?

A:

1. All the VSphereDpeloymentZones having the spec.Server field same as the spec.Server field for the VSphereCluster object will be used to distribute the CP nodes. This association happens automatically when the VSphereCluster and DeploymentZone objects are reconciled.

-1 to this. I think we don't want yet another alternative to make things more confusing and complicated.

2. So the CP nodes will get deployed into the multiple failure domains automatically by the KCP which takes its value from the Cluster object which takes them from the VSphereCluster object - And IIRC, KCP does a round robin on the associated failure domains.

+1 this is how it works (with KCP) and should work with any other ControlPlane implementation, as defined in the contract.
KCP tries to spread the Machines across the available failure domains (For awareness, you're not using KCP in the above example).
To do this, the control plane provider should set .spec.failureDomain on the Machine's it is creating.

[jisnardo]

My kind: RKE2ControlPlane has

  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: VSphereMachineTemplate

And that VSphereMachineTemplate has only vesx01.domain.com

The only failure domain available is fd01, fd02 is related to vesx02.domain.com

Maybe this is not the correct way?

#1720

[chrischdi]

Out of the top of my head: it should Not be Set in VSphereMachineTemplate, but in VSphereFailureDomain / VSphereDeploymentZone instead.

Then using different failuredomains should result in different vcenters.

[jisnardo]

I've reorganized that options (datacenter, datastore, resourcePool, server) between VSphereMachineTemplate, VSphereFailureDomain, VSphereDeploymentZone.

But kind: Cluster has

  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: VSphereCluster
    name: 'capvrke2'

and kind: VSphereCluster must have server

  server: 'vesx01.domain.com'
  thumbprint: '***'

So the resulting VSphereCluster has only available the failure domain fd01 (we use failureDomainSelector: {})

apiVersion: v1
items:
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: VSphereCluster
  metadata:
    annotations:
      meta.helm.sh/release-name: fleetqa-capvrke2
      meta.helm.sh/release-namespace: capvrke2
      objectset.rio.cattle.io/id: default-fleetqa-capvrke2-cattle-fleet-local-system
    creationTimestamp: "2025-09-09T12:43:39Z"
    finalizers:
    - vspherecluster.infrastructure.cluster.x-k8s.io
    generation: 1
    labels:
      app.kubernetes.io/managed-by: Helm
      cluster.x-k8s.io/cluster-name: capvrke2
      objectset.rio.cattle.io/hash: 1f4c422ea07d22e0f588f23dab7efde472efd8af
    name: capvrke2
    namespace: capvrke2
    ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: Cluster
      name: capvrke2
      uid: 2a611148-e61d-4e01-9da5-61fcaa6cd075
    resourceVersion: "410785656"
    uid: bfedd6fa-09b2-467a-afac-cb1fab43036c
  spec:
    controlPlaneEndpoint:
      host: 172.16.1.254
      port: 6443
    failureDomainSelector: {}
    identityRef:
      kind: Secret
      name: capvrke2
    server: vesx01.domain.com
    thumbprint: ***
  status:
    conditions:
    - lastTransitionTime: "2025-09-09T12:43:39Z"
      status: "True"
      type: Ready
    - lastTransitionTime: "2025-09-09T12:43:39Z"
      status: "True"
      type: FailureDomainsAvailable
    - lastTransitionTime: "2025-09-09T12:43:39Z"
      status: "True"
      type: VCenterAvailable
    failureDomains:
      fd01:
        controlPlane: true
    ready: true
    vCenterVersion: 8.0.3
kind: List
metadata:
  resourceVersion: ""

No way to see CP nodes on fd02, that's because we have configured before server at VSphereMachineTemplate for workers.

[chrischdi]

I agree, that definitely is a gap which does not work.

I looked into the vspherecluster controller code and did see that we only add failuredomains pointing to the same vcenter:

cluster-api-provider-vsphere/controllers/vspherecluster_reconciler.go

Line 429 in f3737f1

if zone.Spec.Server != clusterCtx.VSphereCluster.Spec.Server {

I don't know where the pitfalls are if we would just relax that. Pretty sure other things would not work, e.g. cluster modules won't get created and can't be found.

So for me looks like only multiple vcenters are supported when used for different clusters, not in a single cluster.

[chrischdi]

To not forget: thanks for providing all he context and help digging into this!

[jisnardo]

Thanks @chrischdi

Could be possible a release with that feature?

I can do tests and provide feedback.

[chrischdi]

One major issue with that is we won't be able to have e2e coverage for this.

Another one is: while we maybe can easily solve the per-machine issue, making a vspherecluster compatible to multiple vcenters at the same time is way more advanced and seems to be a niche use-case.

Isn't it possible in your case to have a vcenter that spans both other vcenters and layer it on top in that way?

[jisnardo]

We will test Enhanced Linked Mode, a different aproach #1720 @srm09