bug: etcdserver: request is too large when deploying CAPZ

[anmazzotti]

What steps did you take and what happened?

Turtles prints errors when installing CAPZ 1.24.1 (and possibly other versions as well)

apiVersion: v1
kind: Namespace
metadata:
  name: capz-system
---
apiVersion: turtles-capi.cattle.io/v1alpha1
kind: CAPIProvider
metadata:
  name: azure
  namespace: capz-system
spec:
  type: infrastructure
  name: azure

Error:

I0527 12:07:16.695606      15 genericprovider_controller.go:137] "Reconciling provider" controller="ProviderReconciler" controllerGroup="turtles-capi.cattle.io" controllerKind="CAPIProvider" CAPIProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="88ddca35-ffd5-4e7f-a15f-035649e5b9fb"
I0527 12:07:16.695591      15 healthcheck_controller.go:169] "Checking provider health" controller="healthcheck-turtles-capi.cattle.io/v1alpha1, Kind=CAPIProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capz-system/azureserviceoperator-controller-manager" namespace="capz-system" name="azureserviceoperator-controller-manager" reconcileID="14ddc81c-e745-4ceb-b7ed-4235ab0dfa90" provider="CAPIProvider" providerName="azure" deployment="capz-system/azureserviceoperator-controller-manager"
I0527 12:07:16.695843      15 healthcheck_controller.go:169] "Checking provider health" controller="healthcheck-turtles-capi.cattle.io/v1alpha1, Kind=CAPIProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capz-system/capz-controller-manager" namespace="capz-system" name="capz-controller-manager" reconcileID="c0e4552a-aaa8-4bc6-82a0-eacdc710f903" provider="CAPIProvider" providerName="azure" deployment="capz-system/capz-controller-manager"
I0527 12:07:16.698795      15 config.go:157] "Turtles configured to use Rancher default registry for images" controller="ProviderReconciler" controllerGroup="turtles-capi.cattle.io" controllerKind="CAPIProvider" CAPIProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="88ddca35-ffd5-4e7f-a15f-035649e5b9fb"
I0527 12:07:16.803155      15 config.go:157] "Turtles configured to use Rancher default registry for images" controller="ProviderReconciler" controllerGroup="turtles-capi.cattle.io" controllerKind="CAPIProvider" CAPIProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="88ddca35-ffd5-4e7f-a15f-035649e5b9fb"
I0527 12:07:16.803215      15 defaults.go:107] "Checking provider azure last version: latest. Known: false" controller="ProviderReconciler" controllerGroup="turtles-capi.cattle.io" controllerKind="CAPIProvider" CAPIProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="88ddca35-ffd5-4e7f-a15f-035649e5b9fb"
I0527 12:07:16.804917      15 client.go:49] "Updating Secret: capz-system/azure" controller="ProviderReconciler" controllerGroup="turtles-capi.cattle.io" controllerKind="CAPIProvider" CAPIProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="88ddca35-ffd5-4e7f-a15f-035649e5b9fb"
I0527 12:07:16.816355      15 genericprovider_controller.go:476] "Applying provider configuration from cache" controller="ProviderReconciler" controllerGroup="turtles-capi.cattle.io" controllerKind="CAPIProvider" CAPIProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="88ddca35-ffd5-4e7f-a15f-035649e5b9fb"
E0527 12:07:19.234629      15 controller.go:495] "Reconciler error" err="etcdserver: request is too large" controller="ProviderReconciler" controllerGroup="turtles-capi.cattle.io" controllerKind="CAPIProvider" CAPIProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="88ddca35-ffd5-4e7f-a15f-035649e5b9fb"

CAPIProvider is seemingly installed however:

apiVersion: turtles-capi.cattle.io/v1alpha1
kind: CAPIProvider
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"turtles-capi.cattle.io/v1alpha1","kind":"CAPIProvider","metadata":{"annotations":{},"name":"azure","namespace":"capz-system"},"spec":{"name":"azure","type":"infrastructure"}}
    operator.cluster.x-k8s.io/applied-spec-hash: d074d99ffbd0917c7951f956fe181b3d66ad1e3b6ac51923f6d1dbf9c0083aa5
  creationTimestamp: "2026-05-27T12:05:09Z"
  finalizers:
    - provider.cluster.x-k8s.io
  generation: 2
  name: azure
  namespace: capz-system
  resourceVersion: "4410"
  uid: 9cc42bbf-fd74-4566-8587-fc090dba56b1
spec:
  name: azure
  type: infrastructure
  version: v1.24.1
status:
  conditions:
    - lastTransitionTime: "2026-05-27T12:06:16Z"
      message: Deployment has minimum availability.
      observedGeneration: 2
      reason: MinimumReplicasAvailable
      status: "True"
      type: Ready
    - lastTransitionTime: "2026-05-27T12:05:09Z"
      message: Provider is unknown
      observedGeneration: 2
      reason: ProviderUnknown
      status: Unknown
      type: CheckLatestVersionTime
    - lastTransitionTime: "2026-05-27T12:05:09Z"
      message: All preflight checks passed
      observedGeneration: 1
      reason: PreflightChecksPassed
      status: "True"
      type: PreflightCheckPassed
    - lastTransitionTime: "2026-05-27T12:05:15Z"
      message: Provider installed successfully
      observedGeneration: 1
      reason: ProviderInstalled
      status: "True"
      type: ProviderInstalled
    - lastTransitionTime: "2026-05-27T12:05:09Z"
      message: Certificates are now managed by wrangler
      observedGeneration: 1
      reason: CertificatesManaged
      status: "True"
      type: WranglerManagedCertificates
  contract: v1beta1
  installedVersion: v1.24.1
  name: azure
  observedGeneration: 1
  phase: Ready
  variables:
    CLUSTER_TOPOLOGY: "true"
    EXP_AKS_RESOURCE_HEALTH: "true"
    EXP_CLUSTER_RESOURCE_SET: "true"
    EXP_MACHINE_POOL: "true"

The cache configmap also seem to be generated and compressed

kubectl -n capz-system get configmaps infrastructure-azure-v1.24.1:

kind: ConfigMap
metadata:
  annotations:
    provider.cluster.x-k8s.io/compressed: "true"
  creationTimestamp: "2026-05-27T12:05:10Z"
  labels:
    managed-by.operator.cluster.x-k8s.io: "true"
    provider.cluster.x-k8s.io/name: azure
    provider.cluster.x-k8s.io/type: infrastructure
    provider.cluster.x-k8s.io/version: v1.24.1
  name: infrastructure-azure-v1.24.1
  namespace: capz-system
  ownerReferences:
  - apiVersion: turtles-capi.cattle.io/v1alpha1
    kind: CAPIProvider
    name: azure
    uid: 9cc42bbf-fd74-4566-8587-fc090dba56b1
  resourceVersion: "3752"
  uid: 0aa0d251-4674-4106-af88-2a21d1628a06

What did you expect to happen?

Error should not appear

How to reproduce it?

No response

Rancher Turtles version

No response

Anything else you would like to add?

No response

Label(s) to be applied

/kind bug

[anmazzotti]

Also reproduced with v1.24.0.

v1.22.* (Turtles v0.26) and v1.23.* releases of CAPZ do not present this issue. In v1.24 the infrastructure-components.yaml manifest did grow a little, but the cluster-api-operator cache configmap is still within the 1MiB limit:

kubectl -n capz-system get configmaps infrastructure-azure-v1.22.0 -o yaml | wc -c
620434

kubectl -n capz-system get configmaps infrastructure-azure-v1.23.0 -o yaml | wc -c
620427

kubectl -n capz-system get configmaps infrastructure-azure-v1.24.1 -o yaml | wc -c
676364

[anmazzotti]

The problem seems to be the azureserviceoperator-mutating-webhook-configuration when Patching objects from cache.

Made the error a bit more verbose:

E0601 08:17:35.594483     273 controller.go:495] "Reconciler error" err="applying manifests from data: patching object MutatingWebhookConfiguration - azureserviceoperator-mutating-webhook-configuration: etcdserver: request is too large" controller="ProviderReconciler" controllerGroup="turtles-capi.cattle.io" controllerKind="CAPIProvider" CAPIProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="53331ccd-9515-480c-8e2d-7ca3b07754cb"

It looks to be 246261 bytes alone, but on the server is even bigger due to the caBundle for each and every resource, so it's 1981692 stored and that I assume is where the server side apply request fails.

[furkatgofurov7]

@anmazzotti looks like it was already reported in upstream ASO repo: Azure/azure-service-operator#5208 and fixed in Azure/azure-service-operator#5218.

However, the fix is landed in ASO v2.19.0 and CAPZ might need some time to catch up and bump to that ASO version