etcd.local.extraArgs in-place update failed on clusters created with CAPI ≤ v1.11: error "extraArgs name must be unique"

[meghasinghal186]

What steps did you take and what happened?

1. Had a cluster created with CAPI ≤ v1.11
2. Management cluster was upgraded to CAPI v1.12.5
3. Added logic for inplace update for updating spec.clusterConfiguration.etcd.local.extraArgs in KCP specs.
4. Changed spec.clusterConfiguration.etcd.local.extraArgs in the KCP spec (e.g. updated quota-backend-bytes from 6000000000 to 8000000000)
5. In-place update was triggered for existing CP machines. 1st machine got reprovisioned, and next one into Updating state.
6. The KCP reconcile loop repeatedly failed for 2nd CP with the following error on every reconcile, The machine stays stuck in Updating phase indefinitely.

inplace_trigger.go:44] "Triggering in-place update for Machine test-2npxs-tjx8v" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" KubeadmControlPlane="test/test-2npxs" reconcileID="e3a4b919-8045-40de-abf7-469eedc12b41" Cluster="test/test" Machine="test/test-2npxs-tjx8v"
E0803 13:08:34.537165       1 controller.go:474] "Reconciler error" err="failed to complete triggering in-place update for Machine test/test-2npxs-tjx8v: failed to apply KubeadmConfig: KubeadmConfig.bootstrap.cluster.x-k8s.io \"test-2npxs-tjx8v\" is invalid: spec.clusterConfiguration.etcd.local.extraArgs: Invalid value: \"array\": extraArgs name must be unique" controller="kubeadmcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="KubeadmControlPlane" KubeadmControlPlane="test/testl-2npxs" reconcileID="e3a4b919-8045-40de-abf7-469eedc12b41"

During In-Place update, existing spec.clusterConfiguration.etcd.local.extraArgs in old kubeadmconfig is not identified and SSA is trying add a new one with same name.

Checking managedFields on the affected KubeadmConfig shows f:spec is not owned by any field manager after migration:

On the affected KubeadmConfig (old cluster-created before capi1.11, old kubeadmconfig, post-migration):

kubectl get kubeadmconfig <name> -n <namespace> --show-managed-fields -o yaml

  - apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations: {}
        f:labels:
          f:cluster.x-k8s.io/cluster-name: {}
          f:cluster.x-k8s.io/control-plane: {}
          f:cluster.x-k8s.io/control-plane-name: {}
          f:topology.cluster.x-k8s.io/owned: {}
    manager: capi-kubeadmcontrolplane-metadata
    operation: Apply
# f:spec has NO owner — entire spec is orphaned

On a KubeadmConfig from a newly rolled-out node (same cluster, CAPI v1.12):

managedFields:
  - apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:clusterConfiguration:
         ......
          f:etcd: 
            f:local:
              f:extraArgs:
                k:{"name":"auto-compaction-mode","value":"revision"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"auto-compaction-retention","value":"150000"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"quota-backend-bytes","value":"8000000000"}:
                  .: {}
                  f:name: {}
                  f:value: {}
         ....................
        f:joinConfiguration:
          f:controlPlane: {}
          f:nodeRegistration:
            f:imagePullPolicy: {}
            f:kubeletExtraArgs:
              k:{"name":"config","value":"/etc/kubernetes/kubelet/config.yaml"}:
                .: {}
                f:name: {}
                f:value: {}
            f:name: {}
          f:timeouts:
            f:controlPlaneComponentHealthCheckSeconds: {}
        f:postKubeadmCommands: {}
        f:preKubeadmCommands: {}
    manager: capi-kubeadmcontrolplane     # owns f:spec ✅
    operation: Apply

So potentially, capi-kubeadmcontrolplane does not own the old spec and trying to add a new spec.clusterConfiguration.etcd.local.extraArgs?

Steps to reproduce

1. Have a cluster created with CAPI ≤ v1.11
2. Upgrade management cluster to CAPI v1.12+
3. Verify f:spec is orphaned:

   kubectl get kubeadmconfig <name> -n <namespace> --show-managed-fields -o yaml

   Confirm no manager owns f:spec
4. Add in place update for etcd.local.extraArgs
5. Change etcd.local.extraArgs (e.g. update quota-backend-bytes) in KCP spec
6. Observe inplace_trigger.go entering a retry loop with extraArgs name must be unique in capi-kubeadm-control-plane-controller-manager logs

What did you expect to happen?

The in-place update should replace the existing quota-backend-bytes value cleanly — the same as it works correctly on clusters created with CAPI ≥ v1.12, and on new nodes created via rolling replace on the same old cluster.

Confirmed that on newly created clusters (CAPI ≥ v1.12 from day one), repeated in-place updates of the same extraArg (e.g. 150000 → 200000 → 210000) work correctly

Cluster API version

CAPI version: v1.12.5

Kubernetes version

No response

Anything else you would like to add?

Potentially, related to this kubeadmcontrolplane_controller.go#L948 ?

initConfiguration is already handled in inplace_trigger.go#L143

Label(s) to be applied

/kind bug
/area provider/control-plane-kubeadm

[kubernetes-prow]

This issue is currently awaiting triage.

If CAPI contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[sahitya-chandra]

Hi @meghasinghal186, Is it ok if I pick this up, once it gets triaged label by maintainers

[sbueringer]

@meghasinghal186 I'm not sure how/if we can support in-place updates for Machines created with CAPI <= v1.11 in this case. Given that InPlaceUpdate is still an alpha feature it's probably better to live with this limitation compared to the alternative of adding complex logic to try to handle this somehow