Azure CAPIProvider not getting updated after turtles upgrade

[cpinjani]

What steps did you take and what happened?

CAPIProvider versions not getting updated along after turtles is upgrade to v0.18.0-rc.0, probably due to fleet helmops stays in updating state.

0.17.0:

0.18.0-rc.0 (after upgrade)

What did you expect to happen?

CAPIProvider versions should update after turtles upgrade

How to reproduce it?

No response

Rancher Turtles version

Rancher - v2.10.2
Turtles - v0.18.0-rc.0

Anything else you would like to add?

No response

Label(s) to be applied

/kind bug

[Danil-Grigorev]

Helmops controller is intended to be installed on fleet 0.12 version only, which shouldn’t be installed on rancher 2.10. This is caused by issue #1205 and should be fixed when the #1205 is fixed.

[cpinjani]

Validation result on v0.18.0-rc.1, rancher-turtles-cluster-api-operator pod is having OOMKilled issue after upgrade.
The node itself has enough memory.

$ kubectl top node
NAME             CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   
cpinjani-test1   217m         1%       3852Mi          5%

Logs:

I0328 10:31:05.870110       1 webhook.go:163] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=CoreProvider" path="/mutate-operator-cluster-x-k8s-io-v1alpha2-coreprovider"
I0328 10:31:05.870239       1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-operator-cluster-x-k8s-io-v1alpha2-coreprovider"
I0328 10:31:05.870276       1 webhook.go:202] "Registering a validating webhook" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=CoreProvider" path="/validate-operator-cluster-x-k8s-io-v1alpha2-coreprovider"
I0328 10:31:05.870310       1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-operator-cluster-x-k8s-io-v1alpha2-coreprovider"
I0328 10:31:05.870378       1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/convert"
I0328 10:31:05.870400       1 webhook.go:241] "Conversion webhook enabled" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=CoreProvider"
I0328 10:31:05.870419       1 webhook.go:163] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=BootstrapProvider" path="/mutate-operator-cluster-x-k8s-io-v1alpha2-bootstrapprovider"
I0328 10:31:05.870452       1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-operator-cluster-x-k8s-io-v1alpha2-bootstrapprovider"
I0328 10:31:05.870472       1 webhook.go:202] "Registering a validating webhook" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=BootstrapProvider" path="/validate-operator-cluster-x-k8s-io-v1alpha2-bootstrapprovider"
I0328 10:31:05.870504       1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-operator-cluster-x-k8s-io-v1alpha2-bootstrapprovider"
I0328 10:31:05.870526       1 webhook.go:241] "Conversion webhook enabled" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=BootstrapProvider"
I0328 10:31:05.870547       1 webhook.go:163] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=ControlPlaneProvider" path="/mutate-operator-cluster-x-k8s-io-v1alpha2-controlplaneprovider"
I0328 10:31:05.870578       1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-operator-cluster-x-k8s-io-v1alpha2-controlplaneprovider"
I0328 10:31:05.870598       1 webhook.go:202] "Registering a validating webhook" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=ControlPlaneProvider" path="/validate-operator-cluster-x-k8s-io-v1alpha2-controlplaneprovider"
I0328 10:31:05.870626       1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-operator-cluster-x-k8s-io-v1alpha2-controlplaneprovider"
I0328 10:31:05.870647       1 webhook.go:241] "Conversion webhook enabled" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=ControlPlaneProvider"
I0328 10:31:05.870668       1 webhook.go:163] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" path="/mutate-operator-cluster-x-k8s-io-v1alpha2-infrastructureprovider"
I0328 10:31:05.870697       1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-operator-cluster-x-k8s-io-v1alpha2-infrastructureprovider"
I0328 10:31:05.870722       1 webhook.go:202] "Registering a validating webhook" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" path="/validate-operator-cluster-x-k8s-io-v1alpha2-infrastructureprovider"
I0328 10:31:05.870757       1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-operator-cluster-x-k8s-io-v1alpha2-infrastructureprovider"
I0328 10:31:05.870780       1 webhook.go:241] "Conversion webhook enabled" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider"
I0328 10:31:05.870801       1 webhook.go:163] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" path="/mutate-operator-cluster-x-k8s-io-v1alpha2-addonprovider"
I0328 10:31:05.870845       1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-operator-cluster-x-k8s-io-v1alpha2-addonprovider"
I0328 10:31:05.870867       1 webhook.go:202] "Registering a validating webhook" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" path="/validate-operator-cluster-x-k8s-io-v1alpha2-addonprovider"
I0328 10:31:05.870901       1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-operator-cluster-x-k8s-io-v1alpha2-addonprovider"
I0328 10:31:05.870940       1 webhook.go:163] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=IPAMProvider" path="/mutate-operator-cluster-x-k8s-io-v1alpha2-ipamprovider"
I0328 10:31:05.870973       1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-operator-cluster-x-k8s-io-v1alpha2-ipamprovider"
I0328 10:31:05.870994       1 webhook.go:202] "Registering a validating webhook" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=IPAMProvider" path="/validate-operator-cluster-x-k8s-io-v1alpha2-ipamprovider"
I0328 10:31:05.871023       1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-operator-cluster-x-k8s-io-v1alpha2-ipamprovider"
I0328 10:31:05.871056       1 webhook.go:163] "Registering a mutating webhook" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=RuntimeExtensionProvider" path="/mutate-operator-cluster-x-k8s-io-v1alpha2-runtimeextensionprovider"
I0328 10:31:05.871087       1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/mutate-operator-cluster-x-k8s-io-v1alpha2-runtimeextensionprovider"
I0328 10:31:05.871106       1 webhook.go:202] "Registering a validating webhook" logger="controller-runtime.builder" GVK="operator.cluster.x-k8s.io/v1alpha2, Kind=RuntimeExtensionProvider" path="/validate-operator-cluster-x-k8s-io-v1alpha2-runtimeextensionprovider"
I0328 10:31:05.871132       1 server.go:183] "Registering webhook" logger="controller-runtime.webhook" path="/validate-operator-cluster-x-k8s-io-v1alpha2-runtimeextensionprovider"
I0328 10:31:05.871158       1 main.go:202] "starting manager" logger="setup" version=""
I0328 10:31:05.871231       1 server.go:208] "Starting metrics server" logger="controller-runtime.metrics"
I0328 10:31:05.871248       1 server.go:83] "starting server" name="health probe" addr="[::]:9440"
I0328 10:31:05.871651       1 server.go:191] "Starting webhook server" logger="controller-runtime.webhook"
I0328 10:31:05.871681       1 leaderelection.go:254] attempting to acquire leader lease rancher-turtles-system/controller-leader-election-capi-operator...
I0328 10:31:05.871864       1 certwatcher.go:211] "Updated current TLS certificate" logger="controller-runtime.certwatcher"
I0328 10:31:05.871926       1 server.go:242] "Serving webhook server" logger="controller-runtime.webhook" host="" port=9443
I0328 10:31:05.872024       1 certwatcher.go:133] "Starting certificate poll+watcher" logger="controller-runtime.certwatcher" interval="10s"
I0328 10:31:09.567097       1 server.go:247] "Serving metrics server" logger="controller-runtime.metrics" bindAddress=":8443" secure=true
I0328 10:31:24.966771       1 leaderelection.go:268] successfully acquired lease rancher-turtles-system/controller-leader-election-capi-operator
I0328 10:31:24.966957       1 controller.go:175] "Starting EventSource" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" source="kind source: *v1alpha2.CoreProvider"
I0328 10:31:24.966986       1 controller.go:183] "Starting Controller" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider"
I0328 10:31:24.967130       1 controller.go:175] "Starting EventSource" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=CoreProvider" controllerGroup="apps" controllerKind="Deployment" source="kind source: *v1.Deployment"
I0328 10:31:24.967150       1 controller.go:183] "Starting Controller" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=CoreProvider" controllerGroup="apps" controllerKind="Deployment"
I0328 10:31:24.967159       1 controller.go:175] "Starting EventSource" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=IPAMProvider" controllerGroup="apps" controllerKind="Deployment" source="kind source: *v1.Deployment"
I0328 10:31:24.967183       1 controller.go:183] "Starting Controller" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=IPAMProvider" controllerGroup="apps" controllerKind="Deployment"
I0328 10:31:24.967234       1 controller.go:175] "Starting EventSource" controller="runtimeextensionprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="RuntimeExtensionProvider" source="kind source: *v1alpha2.RuntimeExtensionProvider"
I0328 10:31:24.967253       1 controller.go:175] "Starting EventSource" controller="runtimeextensionprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="RuntimeExtensionProvider" source="kind source: *v1alpha2.CoreProvider"
I0328 10:31:24.967266       1 controller.go:183] "Starting Controller" controller="runtimeextensionprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="RuntimeExtensionProvider"
I0328 10:31:24.967277       1 controller.go:175] "Starting EventSource" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" source="kind source: *v1alpha2.BootstrapProvider"
I0328 10:31:24.967301       1 controller.go:175] "Starting EventSource" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" source="kind source: *v1alpha2.CoreProvider"
I0328 10:31:24.967313       1 controller.go:183] "Starting Controller" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider"
I0328 10:31:24.967371       1 controller.go:175] "Starting EventSource" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" source="kind source: *v1alpha2.ControlPlaneProvider"
I0328 10:31:24.967399       1 controller.go:175] "Starting EventSource" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" source="kind source: *v1alpha2.CoreProvider"
I0328 10:31:24.967409       1 controller.go:183] "Starting Controller" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider"
I0328 10:31:24.967446       1 controller.go:175] "Starting EventSource" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" source="kind source: *v1alpha2.InfrastructureProvider"
I0328 10:31:24.967464       1 controller.go:175] "Starting EventSource" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" source="kind source: *v1alpha2.CoreProvider"
I0328 10:31:24.967472       1 controller.go:183] "Starting Controller" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider"
I0328 10:31:24.967513       1 controller.go:175] "Starting EventSource" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" controllerGroup="apps" controllerKind="Deployment" source="kind source: *v1.Deployment"
I0328 10:31:24.967524       1 controller.go:183] "Starting Controller" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" controllerGroup="apps" controllerKind="Deployment"
I0328 10:31:24.967593       1 controller.go:175] "Starting EventSource" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=ControlPlaneProvider" controllerGroup="apps" controllerKind="Deployment" source="kind source: *v1.Deployment"
I0328 10:31:24.967602       1 controller.go:183] "Starting Controller" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=ControlPlaneProvider" controllerGroup="apps" controllerKind="Deployment"
I0328 10:31:24.967642       1 controller.go:175] "Starting EventSource" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=BootstrapProvider" controllerGroup="apps" controllerKind="Deployment" source="kind source: *v1.Deployment"
I0328 10:31:24.967654       1 controller.go:183] "Starting Controller" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=BootstrapProvider" controllerGroup="apps" controllerKind="Deployment"
I0328 10:31:24.967633       1 controller.go:175] "Starting EventSource" controller="ipamprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="IPAMProvider" source="kind source: *v1alpha2.IPAMProvider"
I0328 10:31:24.967669       1 controller.go:175] "Starting EventSource" controller="ipamprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="IPAMProvider" source="kind source: *v1alpha2.CoreProvider"
I0328 10:31:24.967680       1 controller.go:183] "Starting Controller" controller="ipamprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="IPAMProvider"
I0328 10:31:24.967722       1 controller.go:175] "Starting EventSource" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" source="kind source: *v1alpha2.AddonProvider"
I0328 10:31:24.967748       1 controller.go:175] "Starting EventSource" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" source="kind source: *v1alpha2.CoreProvider"
I0328 10:31:24.967759       1 controller.go:183] "Starting Controller" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider"
I0328 10:31:24.967798       1 controller.go:175] "Starting EventSource" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=RuntimeExtensionProvider" controllerGroup="apps" controllerKind="Deployment" source="kind source: *v1.Deployment"
I0328 10:31:24.967809       1 controller.go:183] "Starting Controller" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=RuntimeExtensionProvider" controllerGroup="apps" controllerKind="Deployment"
I0328 10:31:24.967883       1 controller.go:175] "Starting EventSource" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" source="kind source: *v1.Deployment"
I0328 10:31:24.967892       1 controller.go:183] "Starting Controller" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment"
I0328 10:31:24.968270       1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0328 10:31:24.968292       1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0328 10:31:24.970427       1 reflector.go:368] Caches populated for *v1alpha2.CoreProvider from pkg/mod/k8s.io/client-go@v0.31.5/tools/cache/reflector.go:243
I0328 10:31:24.970483       1 reflector.go:368] Caches populated for *v1alpha2.IPAMProvider from pkg/mod/k8s.io/client-go@v0.31.5/tools/cache/reflector.go:243
I0328 10:31:24.970818       1 reflector.go:368] Caches populated for *v1alpha2.InfrastructureProvider from pkg/mod/k8s.io/client-go@v0.31.5/tools/cache/reflector.go:243
I0328 10:31:25.067699       1 reflector.go:368] Caches populated for *v1alpha2.RuntimeExtensionProvider from pkg/mod/k8s.io/client-go@v0.31.5/tools/cache/reflector.go:243
I0328 10:31:25.067732       1 reflector.go:368] Caches populated for *v1alpha2.ControlPlaneProvider from pkg/mod/k8s.io/client-go@v0.31.5/tools/cache/reflector.go:243
I0328 10:31:25.067748       1 reflector.go:368] Caches populated for *v1alpha2.AddonProvider from pkg/mod/k8s.io/client-go@v0.31.5/tools/cache/reflector.go:243
I0328 10:31:25.067784       1 reflector.go:368] Caches populated for *v1alpha2.BootstrapProvider from pkg/mod/k8s.io/client-go@v0.31.5/tools/cache/reflector.go:243
I0328 10:31:25.085252       1 reflector.go:368] Caches populated for *v1.Deployment from pkg/mod/k8s.io/client-go@v0.31.5/tools/cache/reflector.go:243
I0328 10:31:25.170218       1 controller.go:217] "Starting workers" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=BootstrapProvider" controllerGroup="apps" controllerKind="Deployment" worker count=1
I0328 10:31:25.170316       1 controller.go:217] "Starting workers" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=ControlPlaneProvider" controllerGroup="apps" controllerKind="Deployment" worker count=1
I0328 10:31:25.170413       1 controller.go:217] "Starting workers" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" controllerGroup="apps" controllerKind="Deployment" worker count=1
I0328 10:31:25.170470       1 controller.go:217] "Starting workers" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=IPAMProvider" controllerGroup="apps" controllerKind="Deployment" worker count=1
I0328 10:31:25.170518       1 controller.go:217] "Starting workers" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=RuntimeExtensionProvider" controllerGroup="apps" controllerKind="Deployment" worker count=1
I0328 10:31:25.170569       1 controller.go:217] "Starting workers" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" worker count=1
I0328 10:31:25.170633       1 controller.go:217] "Starting workers" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=CoreProvider" controllerGroup="apps" controllerKind="Deployment" worker count=1
I0328 10:31:25.184781       1 controller.go:217] "Starting workers" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" worker count=1
I0328 10:31:25.184788       1 controller.go:217] "Starting workers" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" worker count=1
I0328 10:31:25.184821       1 controller.go:217] "Starting workers" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" worker count=1
I0328 10:31:25.184834       1 genericprovider_controller.go:92] "Reconciling provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="e83b4e36-3b32-43ea-bd01-27adafae98d1"
I0328 10:31:25.184859       1 controller.go:217] "Starting workers" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" worker count=1
I0328 10:31:25.184871       1 controller.go:217] "Starting workers" controller="runtimeextensionprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="RuntimeExtensionProvider" worker count=1
I0328 10:31:25.184855       1 genericprovider_controller.go:92] "Reconciling provider" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="fb8175e8-e597-4e45-846c-543351b508be"
I0328 10:31:25.184860       1 controller.go:217] "Starting workers" controller="ipamprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="IPAMProvider" worker count=1
I0328 10:31:25.184896       1 genericprovider_controller.go:219] "Deleting provider resources" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="e83b4e36-3b32-43ea-bd01-27adafae98d1"
I0328 10:31:25.184893       1 genericprovider_controller.go:92] "Reconciling provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capa-system/aws" namespace="capa-system" name="aws" reconcileID="279dbe76-766e-423b-b669-f00534754e43"
I0328 10:31:25.184922       1 phases.go:602] "Deleting provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="e83b4e36-3b32-43ea-bd01-27adafae98d1"
I0328 10:31:25.184837       1 genericprovider_controller.go:92] "Reconciling provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="capi-kubeadm-control-plane-system/kubeadm" namespace="capi-kubeadm-control-plane-system" name="kubeadm" reconcileID="6ea08a9d-2400-400d-a404-5f5dab7ee3f5"
I0328 10:31:25.184940       1 genericprovider_controller.go:219] "Deleting provider resources" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="fb8175e8-e597-4e45-846c-543351b508be"
I0328 10:31:25.184957       1 phases.go:602] "Deleting provider" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="fb8175e8-e597-4e45-846c-543351b508be"
I0328 10:31:25.184916       1 controller.go:217] "Starting workers" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" worker count=1
I0328 10:31:25.185035       1 genericprovider_controller.go:92] "Reconciling provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="capi-kubeadm-bootstrap-system/kubeadm" namespace="capi-kubeadm-bootstrap-system" name="kubeadm" reconcileID="d1534a25-8e81-4f10-82c1-bfe9620c2241"
I0328 10:31:25.188319       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="capi-kubeadm-control-plane-system/kubeadm" namespace="capi-kubeadm-control-plane-system" name="kubeadm" reconcileID="6ea08a9d-2400-400d-a404-5f5dab7ee3f5"
I0328 10:31:25.188332       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="capi-kubeadm-bootstrap-system/kubeadm" namespace="capi-kubeadm-bootstrap-system" name="kubeadm" reconcileID="d1534a25-8e81-4f10-82c1-bfe9620c2241"
I0328 10:31:25.188427       1 preflight_checks.go:58] "Performing preflight checks" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capa-system/aws" namespace="capa-system" name="aws" reconcileID="279dbe76-766e-423b-b669-f00534754e43"
I0328 10:31:25.188858       1 genericprovider_controller.go:92] "Reconciling provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="23681792-4c03-4ae4-92f4-02e7a56fa590"
I0328 10:31:25.188887       1 genericprovider_controller.go:219] "Deleting provider resources" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="23681792-4c03-4ae4-92f4-02e7a56fa590"
I0328 10:31:25.188895       1 phases.go:602] "Deleting provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="23681792-4c03-4ae4-92f4-02e7a56fa590"
I0328 10:31:25.188917       1 genericprovider_controller.go:92] "Reconciling provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="c57bae1f-20fb-4716-8765-f7fcd5f9f561"
I0328 10:31:25.188959       1 genericprovider_controller.go:219] "Deleting provider resources" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="c57bae1f-20fb-4716-8765-f7fcd5f9f561"
I0328 10:31:25.188972       1 phases.go:602] "Deleting provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="c57bae1f-20fb-4716-8765-f7fcd5f9f561"
I0328 10:31:25.267140       1 preflight_checks.go:199] "Preflight checks passed" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capa-system/aws" namespace="capa-system" name="aws" reconcileID="279dbe76-766e-423b-b669-f00534754e43"
I0328 10:31:26.168840       1 genericprovider_controller.go:92] "Reconciling provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="8fefe20b-6af3-4104-be8a-feac8ed988f9"
I0328 10:31:26.487102       1 genericprovider_controller.go:92] "Reconciling provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="de353258-b226-414a-a7bf-6ac5b781b1c2"
I0328 10:31:26.772104       1 genericprovider_controller.go:92] "Reconciling provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="capi-kubeadm-control-plane-system/kubeadm" namespace="capi-kubeadm-control-plane-system" name="kubeadm" reconcileID="3e78f28a-c83c-43e4-81cf-93cb8fb24bc1"
I0328 10:31:26.784593       1 preflight_checks.go:58] "Performing preflight checks" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="capi-kubeadm-control-plane-system/kubeadm" namespace="capi-kubeadm-control-plane-system" name="kubeadm" reconcileID="3e78f28a-c83c-43e4-81cf-93cb8fb24bc1"
I0328 10:31:26.786369       1 preflight_checks.go:199] "Preflight checks passed" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="capi-kubeadm-control-plane-system/kubeadm" namespace="capi-kubeadm-control-plane-system" name="kubeadm" reconcileID="3e78f28a-c83c-43e4-81cf-93cb8fb24bc1"
I0328 10:31:26.805526       1 genericprovider_controller.go:92] "Reconciling provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="81cfca17-23c8-43fc-a8ab-df8270f2e5dc"
I0328 10:31:27.084719       1 genericprovider_controller.go:92] "Reconciling provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="capi-kubeadm-bootstrap-system/kubeadm" namespace="capi-kubeadm-bootstrap-system" name="kubeadm" reconcileID="bdb81654-0094-4a4d-8710-8454860e2bbb"
I0328 10:31:27.086055       1 preflight_checks.go:58] "Performing preflight checks" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="capi-kubeadm-bootstrap-system/kubeadm" namespace="capi-kubeadm-bootstrap-system" name="kubeadm" reconcileID="bdb81654-0094-4a4d-8710-8454860e2bbb"
I0328 10:31:27.267355       1 preflight_checks.go:199] "Preflight checks passed" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="capi-kubeadm-bootstrap-system/kubeadm" namespace="capi-kubeadm-bootstrap-system" name="kubeadm" reconcileID="bdb81654-0094-4a4d-8710-8454860e2bbb"
I0328 10:31:27.367136       1 warning_handler.go:65] "metadata.finalizers: \"provider.cluster.x-k8s.io\": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers" logger="KubeAPIWarningLogger"
I0328 10:31:27.667187       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="69d618b1-a449-4d4d-8908-bd624e60e6ec"
E0328 10:31:27.667267       1 controller.go:316] "Reconciler error" err="Deployment.apps \"caapf-controller-manager\" not found" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="69d618b1-a449-4d4d-8908-bd624e60e6ec"
I0328 10:31:27.685247       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="542f07fa-d597-4036-af67-763fff9d32b2"
E0328 10:31:27.685295       1 controller.go:316] "Reconciler error" err="Deployment.apps \"caapf-controller-manager\" not found" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="542f07fa-d597-4036-af67-763fff9d32b2"
I0328 10:31:27.867270       1 genericprovider_controller.go:92] "Reconciling provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="d49a27b3-305b-41b1-8cad-89bc2acd326e"
I0328 10:31:27.867313       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="95c570b0-beff-4d46-aedb-915b1908d9f2"
E0328 10:31:27.867365       1 controller.go:316] "Reconciler error" err="Deployment.apps \"caapf-controller-manager\" not found" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="95c570b0-beff-4d46-aedb-915b1908d9f2"
I0328 10:31:27.869060       1 preflight_checks.go:58] "Performing preflight checks" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="d49a27b3-305b-41b1-8cad-89bc2acd326e"
I0328 10:31:27.870830       1 preflight_checks.go:199] "Preflight checks passed" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="d49a27b3-305b-41b1-8cad-89bc2acd326e"
I0328 10:31:27.966840       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="3f904978-8d80-48fe-8631-668ee55a5cf8"
E0328 10:31:27.966908       1 controller.go:316] "Reconciler error" err="Deployment.apps \"caapf-controller-manager\" not found" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="3f904978-8d80-48fe-8631-668ee55a5cf8"
I0328 10:31:28.067088       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="76cb0421-a3ae-4c91-937e-83ad6098763a"
E0328 10:31:28.067158       1 controller.go:316] "Reconciler error" err="Deployment.apps \"caapf-controller-manager\" not found" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="76cb0421-a3ae-4c91-937e-83ad6098763a"
I0328 10:31:28.166922       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="6a347e7a-2d6e-4db3-bc37-44e77268dc65"
E0328 10:31:28.167098       1 controller.go:316] "Reconciler error" err="Deployment.apps \"caapf-controller-manager\" not found" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="6a347e7a-2d6e-4db3-bc37-44e77268dc65"
I0328 10:31:28.366943       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="ff5829c8-b37a-41cb-88c7-6bf115641d56"
E0328 10:31:28.367343       1 controller.go:316] "Reconciler error" err="Deployment.apps \"caapf-controller-manager\" not found" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="ff5829c8-b37a-41cb-88c7-6bf115641d56"
I0328 10:31:28.766934       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="fbe684cb-e49d-4dfd-846e-d01a062441a6"
E0328 10:31:28.767028       1 controller.go:316] "Reconciler error" err="Deployment.apps \"caapf-controller-manager\" not found" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="fbe684cb-e49d-4dfd-846e-d01a062441a6"
I0328 10:31:28.868161       1 phases.go:182] "Loading provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capa-system/aws" namespace="capa-system" name="aws" reconcileID="279dbe76-766e-423b-b669-f00534754e43"
I0328 10:31:28.967296       1 genericprovider_controller.go:92] "Reconciling provider" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="2160b45f-9faa-4fed-8bd2-4be90a31df24"
I0328 10:31:29.369620       1 genericprovider_controller.go:92] "Reconciling provider" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="e5e2eea1-e087-4b2b-b2e9-478f31c73dd8"
I0328 10:31:29.386568       1 phases.go:448] "Fetching provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capa-system/aws" namespace="capa-system" name="aws" reconcileID="279dbe76-766e-423b-b669-f00534754e43"
I0328 10:31:29.467455       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="326318ce-b60f-42a0-b853-5e3bc62d9b3e"
E0328 10:31:29.467560       1 controller.go:316] "Reconciler error" err="Deployment.apps \"caapf-controller-manager\" not found" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="326318ce-b60f-42a0-b853-5e3bc62d9b3e"
I0328 10:31:29.584778       1 genericprovider_controller.go:92] "Reconciling provider" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="e94f43d5-c680-48f9-bd7d-6faa8009fec9"
I0328 10:31:29.584837       1 preflight_checks.go:58] "Performing preflight checks" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="e94f43d5-c680-48f9-bd7d-6faa8009fec9"
I0328 10:31:29.667049       1 preflight_checks.go:199] "Preflight checks passed" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="e94f43d5-c680-48f9-bd7d-6faa8009fec9"
I0328 10:31:29.684792       1 phases.go:261] "No configuration secret was specified" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="e94f43d5-c680-48f9-bd7d-6faa8009fec9"
I0328 10:31:30.368231       1 phases.go:279] "Custom fetch configuration url was provided" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="capi-kubeadm-control-plane-system/kubeadm" namespace="capi-kubeadm-control-plane-system" name="kubeadm" reconcileID="3e78f28a-c83c-43e4-81cf-93cb8fb24bc1"
I0328 10:31:30.566948       1 manifests_downloader.go:76] "Downloading provider manifests" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="d49a27b3-305b-41b1-8cad-89bc2acd326e"
I0328 10:31:30.766928       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="d051739b-669b-42a6-8efd-822334af573b"
E0328 10:31:30.767020       1 controller.go:316] "Reconciler error" err="Deployment.apps \"caapf-controller-manager\" not found" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="d051739b-669b-42a6-8efd-822334af573b"
I0328 10:31:31.086030       1 phases.go:182] "Loading provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="capi-kubeadm-control-plane-system/kubeadm" namespace="capi-kubeadm-control-plane-system" name="kubeadm" reconcileID="3e78f28a-c83c-43e4-81cf-93cb8fb24bc1"
I0328 10:31:31.185089       1 phases.go:448] "Fetching provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="capi-kubeadm-control-plane-system/kubeadm" namespace="capi-kubeadm-control-plane-system" name="kubeadm" reconcileID="3e78f28a-c83c-43e4-81cf-93cb8fb24bc1"
I0328 10:31:31.268232       1 phases.go:279] "Custom fetch configuration url was provided" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="capi-kubeadm-bootstrap-system/kubeadm" namespace="capi-kubeadm-bootstrap-system" name="kubeadm" reconcileID="bdb81654-0094-4a4d-8710-8454860e2bbb"
I0328 10:31:31.985224       1 phases.go:182] "Loading provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="capi-kubeadm-bootstrap-system/kubeadm" namespace="capi-kubeadm-bootstrap-system" name="kubeadm" reconcileID="bdb81654-0094-4a4d-8710-8454860e2bbb"
I0328 10:31:32.284809       1 phases.go:448] "Fetching provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="capi-kubeadm-bootstrap-system/kubeadm" namespace="capi-kubeadm-bootstrap-system" name="kubeadm" reconcileID="bdb81654-0094-4a4d-8710-8454860e2bbb"
I0328 10:31:32.747349       1 phases.go:182] "Loading provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="d49a27b3-305b-41b1-8cad-89bc2acd326e"
I0328 10:31:32.769698       1 phases.go:448] "Fetching provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="d49a27b3-305b-41b1-8cad-89bc2acd326e"
I0328 10:31:33.366879       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="c87c6f62-9f2d-42f7-a81f-1eaa9f9de7e1"
E0328 10:31:33.366971       1 controller.go:316] "Reconciler error" err="Deployment.apps \"caapf-controller-manager\" not found" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="c87c6f62-9f2d-42f7-a81f-1eaa9f9de7e1"
I0328 10:31:34.169553       1 manifests_downloader.go:76] "Downloading provider manifests" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="e94f43d5-c680-48f9-bd7d-6faa8009fec9"
I0328 10:31:35.784880       1 phases.go:534] "Installing provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="d49a27b3-305b-41b1-8cad-89bc2acd326e"
I0328 10:31:35.967682       1 phases.go:182] "Loading provider" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="e94f43d5-c680-48f9-bd7d-6faa8009fec9"
I0328 10:31:36.281601       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="486c3857-244f-4122-b72a-3b47eb3e8240"
I0328 10:31:36.289108       1 phases.go:545] "Provider successfully installed" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="d49a27b3-305b-41b1-8cad-89bc2acd326e"
I0328 10:31:36.292782       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="9e358be9-f621-486f-bd20-378f0c04863b"
I0328 10:31:36.367313       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="2d6cc4c3-03d6-4bac-801c-9d01df3d1812"
I0328 10:31:36.567885       1 phases.go:448] "Fetching provider" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="e94f43d5-c680-48f9-bd7d-6faa8009fec9"
I0328 10:31:36.974323       1 genericprovider_controller.go:92] "Reconciling provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="86055e93-1411-4a6c-8a26-1618698ba5c0"
I0328 10:31:37.068347       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="86055e93-1411-4a6c-8a26-1618698ba5c0"
I0328 10:31:37.167336       1 genericprovider_controller.go:92] "Reconciling provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="d1e270d2-e35e-4ca1-9338-598b26a57a9e"
I0328 10:31:37.268496       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="d1e270d2-e35e-4ca1-9338-598b26a57a9e"
I0328 10:31:38.567095       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="12dbce64-1206-491e-93e8-6f9c33e2b066"
I0328 10:31:38.767233       1 genericprovider_controller.go:92] "Reconciling provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="1709bed6-eee1-44d3-b1ed-64ea3e28deea"
I0328 10:31:38.886191       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="1709bed6-eee1-44d3-b1ed-64ea3e28deea"
I0328 10:31:39.870801       1 phases.go:534] "Installing provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="capi-kubeadm-control-plane-system/kubeadm" namespace="capi-kubeadm-control-plane-system" name="kubeadm" reconcileID="3e78f28a-c83c-43e4-81cf-93cb8fb24bc1"
I0328 10:31:39.966825       1 phases.go:534] "Installing provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="capi-kubeadm-bootstrap-system/kubeadm" namespace="capi-kubeadm-bootstrap-system" name="kubeadm" reconcileID="bdb81654-0094-4a4d-8710-8454860e2bbb"
I0328 10:31:43.084945       1 phases.go:545] "Provider successfully installed" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="capi-kubeadm-bootstrap-system/kubeadm" namespace="capi-kubeadm-bootstrap-system" name="kubeadm" reconcileID="bdb81654-0094-4a4d-8710-8454860e2bbb"
I0328 10:31:43.288846       1 genericprovider_controller.go:92] "Reconciling provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="da5311f6-40b3-4d66-9dcd-92fe3cdad28b"
I0328 10:31:43.467745       1 phases.go:545] "Provider successfully installed" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="capi-kubeadm-control-plane-system/kubeadm" namespace="capi-kubeadm-control-plane-system" name="kubeadm" reconcileID="3e78f28a-c83c-43e4-81cf-93cb8fb24bc1"
I0328 10:31:43.567557       1 genericprovider_controller.go:92] "Reconciling provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="capi-kubeadm-bootstrap-system/kubeadm" namespace="capi-kubeadm-bootstrap-system" name="kubeadm" reconcileID="044b279c-d15f-48bf-8156-235ed7e3f91a"
2025/03/28 10:31:43 http: TLS handshake error from 10.42.0.1:48538: EOF
I0328 10:31:43.667042       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="capi-kubeadm-bootstrap-system/kubeadm" namespace="capi-kubeadm-bootstrap-system" name="kubeadm" reconcileID="044b279c-d15f-48bf-8156-235ed7e3f91a"
I0328 10:31:43.667528       1 genericprovider_controller.go:92] "Reconciling provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="1099426e-f3b7-4366-8a4e-7e089ec32a96"
I0328 10:31:43.668744       1 preflight_checks.go:58] "Performing preflight checks" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="1099426e-f3b7-4366-8a4e-7e089ec32a96"
I0328 10:31:43.768607       1 preflight_checks.go:185] "Waiting for the core provider to be installed." controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="1099426e-f3b7-4366-8a4e-7e089ec32a96"
I0328 10:31:43.788297       1 genericprovider_controller.go:92] "Reconciling provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="368f2fc9-6fef-438c-aa52-9f1bd072df7f"
I0328 10:31:43.967054       1 genericprovider_controller.go:92] "Reconciling provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="capi-kubeadm-control-plane-system/kubeadm" namespace="capi-kubeadm-control-plane-system" name="kubeadm" reconcileID="f91a3613-be19-40b5-a54e-0214a2a2df71"
I0328 10:31:43.968535       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="capi-kubeadm-control-plane-system/kubeadm" namespace="capi-kubeadm-control-plane-system" name="kubeadm" reconcileID="f91a3613-be19-40b5-a54e-0214a2a2df71"
I0328 10:31:43.968851       1 genericprovider_controller.go:92] "Reconciling provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="aecb3a0b-6808-4c4e-91c7-7167b9f9ecf0"
I0328 10:31:43.984511       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="22cb8886-8090-4266-b599-efcb4daa04e8"
I0328 10:31:44.067015       1 preflight_checks.go:58] "Performing preflight checks" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="aecb3a0b-6808-4c4e-91c7-7167b9f9ecf0"
I0328 10:31:44.070190       1 preflight_checks.go:185] "Waiting for the core provider to be installed." controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="aecb3a0b-6808-4c4e-91c7-7167b9f9ecf0"
I0328 10:31:45.287892       1 manifests_downloader.go:76] "Downloading provider manifests" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="1099426e-f3b7-4366-8a4e-7e089ec32a96"
I0328 10:31:45.567242       1 phases.go:534] "Installing provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capa-system/aws" namespace="capa-system" name="aws" reconcileID="279dbe76-766e-423b-b669-f00534754e43"
I0328 10:31:45.767359       1 manifests_downloader.go:76] "Downloading provider manifests" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="aecb3a0b-6808-4c4e-91c7-7167b9f9ecf0"
I0328 10:31:45.785676       1 phases.go:182] "Loading provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="1099426e-f3b7-4366-8a4e-7e089ec32a96"
I0328 10:31:45.886269       1 phases.go:534] "Installing provider" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="e94f43d5-c680-48f9-bd7d-6faa8009fec9"
I0328 10:31:46.067475       1 phases.go:448] "Fetching provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="1099426e-f3b7-4366-8a4e-7e089ec32a96"
I0328 10:31:46.267414       1 phases.go:182] "Loading provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="aecb3a0b-6808-4c4e-91c7-7167b9f9ecf0"
I0328 10:31:46.367618       1 phases.go:448] "Fetching provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="aecb3a0b-6808-4c4e-91c7-7167b9f9ecf0"
I0328 10:31:46.612782       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=AddonProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rancher-turtles-system/caapf-controller-manager" namespace="rancher-turtles-system" name="caapf-controller-manager" reconcileID="32bd1d33-9186-46c7-afbb-2394c7878417"
I0328 10:31:46.767599       1 genericprovider_controller.go:92] "Reconciling provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="7643ef54-75a9-4a54-8ac8-18715612194a"
I0328 10:31:46.769032       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="rancher-turtles-system/fleet" namespace="rancher-turtles-system" name="fleet" reconcileID="7643ef54-75a9-4a54-8ac8-18715612194a"
I0328 10:31:47.067156       1 phases.go:534] "Installing provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="1099426e-f3b7-4366-8a4e-7e089ec32a96"
I0328 10:31:48.081176       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=BootstrapProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rke2-bootstrap-system/rke2-bootstrap-controller-manager" namespace="rke2-bootstrap-system" name="rke2-bootstrap-controller-manager" reconcileID="c8cb2602-d2eb-442c-95d7-9a8ab09085ae"
I0328 10:31:48.085209       1 phases.go:534] "Installing provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="aecb3a0b-6808-4c4e-91c7-7167b9f9ecf0"
I0328 10:31:48.167264       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=BootstrapProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rke2-bootstrap-system/rke2-bootstrap-controller-manager" namespace="rke2-bootstrap-system" name="rke2-bootstrap-controller-manager" reconcileID="0c5a6145-3a0c-4fbb-ad48-b4221337fd43"
I0328 10:31:48.174644       1 phases.go:545] "Provider successfully installed" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="1099426e-f3b7-4366-8a4e-7e089ec32a96"
I0328 10:31:48.372430       1 genericprovider_controller.go:92] "Reconciling provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="532120d9-f732-441f-860d-fea4f01171e1"
I0328 10:31:48.373683       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="532120d9-f732-441f-860d-fea4f01171e1"
I0328 10:31:49.090771       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=ControlPlaneProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rke2-control-plane-system/rke2-control-plane-controller-manager" namespace="rke2-control-plane-system" name="rke2-control-plane-controller-manager" reconcileID="be253a81-5d39-41f3-bfef-3b58a8f1185e"
I0328 10:31:49.167122       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=ControlPlaneProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rke2-control-plane-system/rke2-control-plane-controller-manager" namespace="rke2-control-plane-system" name="rke2-control-plane-controller-manager" reconcileID="ede48ca3-751c-410a-b4f6-1239217fd262"
I0328 10:31:49.167144       1 phases.go:545] "Provider successfully installed" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="aecb3a0b-6808-4c4e-91c7-7167b9f9ecf0"
I0328 10:31:49.184808       1 genericprovider_controller.go:92] "Reconciling provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="ba4f6a2f-8943-4919-a0fa-278f9ab28cfd"
I0328 10:31:49.185875       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="ba4f6a2f-8943-4919-a0fa-278f9ab28cfd"
I0328 10:31:49.267166       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=CoreProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capi-system/capi-controller-manager" namespace="capi-system" name="capi-controller-manager" reconcileID="5af40448-dd97-4ace-aea4-6d587c5d3ce0"
I0328 10:31:49.267308       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=CoreProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capi-system/capi-controller-manager" namespace="capi-system" name="capi-controller-manager" reconcileID="af2ce008-7d18-40a1-ae5f-a90a65acfc60"
I0328 10:31:49.279365       1 phases.go:545] "Provider successfully installed" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="e94f43d5-c680-48f9-bd7d-6faa8009fec9"
I0328 10:31:49.376916       1 genericprovider_controller.go:92] "Reconciling provider" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="278e7f16-69da-4eab-827b-f4e3dcf0c8cf"
I0328 10:31:49.376974       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="278e7f16-69da-4eab-827b-f4e3dcf0c8cf"
I0328 10:31:49.382413       1 genericprovider_controller.go:92] "Reconciling provider" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="a6c74068-adaf-4922-b549-65141baf87b4"
I0328 10:31:49.382467       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="a6c74068-adaf-4922-b549-65141baf87b4"
I0328 10:31:49.385578       1 genericprovider_controller.go:92] "Reconciling provider" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="7cacd3ec-0679-4704-b1e2-28f8aa3bbcaf"
I0328 10:31:49.385618       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="7cacd3ec-0679-4704-b1e2-28f8aa3bbcaf"
I0328 10:31:50.624869       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=CoreProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capi-system/capi-controller-manager" namespace="capi-system" name="capi-controller-manager" reconcileID="9d031d76-58d4-4a82-981c-45d4445c4e90"
I0328 10:31:50.629069       1 genericprovider_controller.go:92] "Reconciling provider" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="2efd244e-90bf-45d2-a441-5ef415e156b8"
I0328 10:31:50.629080       1 genericprovider_controller.go:92] "Reconciling provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="1778cade-ed5a-4b34-b7a1-4b81f86aec82"
I0328 10:31:50.629113       1 genericprovider_controller.go:92] "Reconciling provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="2b2b33bf-7334-42b3-96e3-7907aee9a401"
I0328 10:31:50.629118       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="2efd244e-90bf-45d2-a441-5ef415e156b8"
I0328 10:31:50.630401       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="1778cade-ed5a-4b34-b7a1-4b81f86aec82"
I0328 10:31:50.630423       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="2b2b33bf-7334-42b3-96e3-7907aee9a401"
I0328 10:31:53.081573       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=BootstrapProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rke2-bootstrap-system/rke2-bootstrap-controller-manager" namespace="rke2-bootstrap-system" name="rke2-bootstrap-controller-manager" reconcileID="db2d78d6-0aea-43ea-89c2-47c2b373a952"
I0328 10:31:53.089684       1 genericprovider_controller.go:92] "Reconciling provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="840e318f-6fe5-4e0c-a1ac-036cccbf210d"
I0328 10:31:53.091067       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="840e318f-6fe5-4e0c-a1ac-036cccbf210d"
I0328 10:31:53.643400       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=BootstrapProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rke2-bootstrap-system/rke2-bootstrap-controller-manager" namespace="rke2-bootstrap-system" name="rke2-bootstrap-controller-manager" reconcileID="cf6cbfd7-385f-434d-977f-500aa10855e7"
I0328 10:31:53.649938       1 genericprovider_controller.go:92] "Reconciling provider" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="b9579445-d66c-4c17-8bd9-a269cd1a3ce0"
I0328 10:31:53.651362       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="rke2-bootstrap-system/rke2" namespace="rke2-bootstrap-system" name="rke2" reconcileID="b9579445-d66c-4c17-8bd9-a269cd1a3ce0"
I0328 10:31:54.091056       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=ControlPlaneProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rke2-control-plane-system/rke2-control-plane-controller-manager" namespace="rke2-control-plane-system" name="rke2-control-plane-controller-manager" reconcileID="12ea4d73-e031-4edb-be0e-a665737b0224"
I0328 10:31:54.267473       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=CoreProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capi-system/capi-controller-manager" namespace="capi-system" name="capi-controller-manager" reconcileID="6b71e49d-07cc-46ef-bdeb-a88b4c37da6d"
I0328 10:31:54.494578       1 genericprovider_controller.go:92] "Reconciling provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="f75fe2a6-9e0b-4838-8a39-487092c7bc01"
I0328 10:31:54.496027       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="f75fe2a6-9e0b-4838-8a39-487092c7bc01"
I0328 10:31:55.655509       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=ControlPlaneProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rke2-control-plane-system/rke2-control-plane-controller-manager" namespace="rke2-control-plane-system" name="rke2-control-plane-controller-manager" reconcileID="e9161571-73e1-4864-acc8-379d946abde7"
I0328 10:31:55.661433       1 genericprovider_controller.go:92] "Reconciling provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="8761a0f6-2256-4aff-87eb-14e754e72dba"
I0328 10:31:55.662467       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="8761a0f6-2256-4aff-87eb-14e754e72dba"
I0328 10:31:58.094080       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=BootstrapProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rke2-bootstrap-system/rke2-bootstrap-controller-manager" namespace="rke2-bootstrap-system" name="rke2-bootstrap-controller-manager" reconcileID="866030d6-4474-49ab-9abe-77763f5de0ad"
I0328 10:31:59.499775       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=ControlPlaneProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="rke2-control-plane-system/rke2-control-plane-controller-manager" namespace="rke2-control-plane-system" name="rke2-control-plane-controller-manager" reconcileID="6f9ccde5-4eac-48a3-8cc6-dda9901dce5d"
E0328 10:32:28.783950       1 controller.go:316] "Reconciler error" err="action failed after 10 attempts: failed to patch provider object: CustomResourceDefinition.apiextensions.k8s.io \"awsclustercontrolleridentities.infrastructure.cluster.x-k8s.io\" is invalid: spec.conversion.webhookClientConfig.caBundle: Invalid value: []byte{0xa}: unable to load root certificates: unable to parse bytes as PEM block" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capa-system/aws" namespace="capa-system" name="aws" reconcileID="279dbe76-766e-423b-b669-f00534754e43"
I0328 10:32:28.784025       1 genericprovider_controller.go:92] "Reconciling provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capd-system/docker" namespace="capd-system" name="docker" reconcileID="f3e485d6-0f3e-4982-8b27-4f933a759329"
I0328 10:32:28.785221       1 preflight_checks.go:58] "Performing preflight checks" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capd-system/docker" namespace="capd-system" name="docker" reconcileID="f3e485d6-0f3e-4982-8b27-4f933a759329"
I0328 10:32:28.786853       1 preflight_checks.go:199] "Preflight checks passed" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capd-system/docker" namespace="capd-system" name="docker" reconcileID="f3e485d6-0f3e-4982-8b27-4f933a759329"
I0328 10:32:29.292437       1 phases.go:182] "Loading provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capd-system/docker" namespace="capd-system" name="docker" reconcileID="f3e485d6-0f3e-4982-8b27-4f933a759329"
I0328 10:32:29.385061       1 phases.go:448] "Fetching provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capd-system/docker" namespace="capd-system" name="docker" reconcileID="f3e485d6-0f3e-4982-8b27-4f933a759329"
I0328 10:32:29.789246       1 phases.go:534] "Installing provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capd-system/docker" namespace="capd-system" name="docker" reconcileID="f3e485d6-0f3e-4982-8b27-4f933a759329"
I0328 10:32:30.211598       1 phases.go:545] "Provider successfully installed" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capd-system/docker" namespace="capd-system" name="docker" reconcileID="f3e485d6-0f3e-4982-8b27-4f933a759329"
I0328 10:32:30.221232       1 genericprovider_controller.go:92] "Reconciling provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capg-system/gcp" namespace="capg-system" name="gcp" reconcileID="633b6e54-c26a-41e3-a22e-6cc63afb14ac"
I0328 10:32:30.222463       1 preflight_checks.go:58] "Performing preflight checks" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capg-system/gcp" namespace="capg-system" name="gcp" reconcileID="633b6e54-c26a-41e3-a22e-6cc63afb14ac"
I0328 10:32:30.224263       1 preflight_checks.go:199] "Preflight checks passed" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capg-system/gcp" namespace="capg-system" name="gcp" reconcileID="633b6e54-c26a-41e3-a22e-6cc63afb14ac"
I0328 10:32:30.888714       1 manifests_downloader.go:76] "Downloading provider manifests" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capg-system/gcp" namespace="capg-system" name="gcp" reconcileID="633b6e54-c26a-41e3-a22e-6cc63afb14ac"
I0328 10:32:31.460485       1 phases.go:182] "Loading provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capg-system/gcp" namespace="capg-system" name="gcp" reconcileID="633b6e54-c26a-41e3-a22e-6cc63afb14ac"
I0328 10:32:31.465547       1 phases.go:448] "Fetching provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capg-system/gcp" namespace="capg-system" name="gcp" reconcileID="633b6e54-c26a-41e3-a22e-6cc63afb14ac"
I0328 10:32:31.989050       1 phases.go:508] "Version changes detected, updating existing components" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capg-system/gcp" namespace="capg-system" name="gcp" reconcileID="633b6e54-c26a-41e3-a22e-6cc63afb14ac"
E0328 10:32:32.598778       1 controller.go:316] "Reconciler error" err="config map not found" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capg-system/gcp" namespace="capg-system" name="gcp" reconcileID="633b6e54-c26a-41e3-a22e-6cc63afb14ac"
I0328 10:32:32.598833       1 genericprovider_controller.go:92] "Reconciling provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="43e4d6a2-c4b6-4b83-8ff9-67377ce1ac07"
I0328 10:32:32.599954       1 preflight_checks.go:58] "Performing preflight checks" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="43e4d6a2-c4b6-4b83-8ff9-67377ce1ac07"
I0328 10:32:32.785000       1 preflight_checks.go:199] "Preflight checks passed" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="43e4d6a2-c4b6-4b83-8ff9-67377ce1ac07"
I0328 10:32:33.289298       1 manifests_downloader.go:76] "Downloading provider manifests" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="43e4d6a2-c4b6-4b83-8ff9-67377ce1ac07"
I0328 10:32:34.985224       1 phases.go:182] "Loading provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="43e4d6a2-c4b6-4b83-8ff9-67377ce1ac07"
I0328 10:32:35.487671       1 phases.go:448] "Fetching provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="43e4d6a2-c4b6-4b83-8ff9-67377ce1ac07"
I0328 10:32:48.887885       1 phases.go:508] "Version changes detected, updating existing components" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="43e4d6a2-c4b6-4b83-8ff9-67377ce1ac07"
E0328 10:32:49.301316       1 controller.go:316] "Reconciler error" err="config map not found" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="43e4d6a2-c4b6-4b83-8ff9-67377ce1ac07"
I0328 10:32:49.301368       1 genericprovider_controller.go:92] "Reconciling provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capa-system/aws" namespace="capa-system" name="aws" reconcileID="cfbcc867-7dfc-494a-9fce-cb55edd927d1"
I0328 10:32:49.302558       1 preflight_checks.go:58] "Performing preflight checks" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capa-system/aws" namespace="capa-system" name="aws" reconcileID="cfbcc867-7dfc-494a-9fce-cb55edd927d1"
I0328 10:32:49.304070       1 preflight_checks.go:199] "Preflight checks passed" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capa-system/aws" namespace="capa-system" name="aws" reconcileID="cfbcc867-7dfc-494a-9fce-cb55edd927d1"
I0328 10:32:49.687213       1 manifests_downloader.go:76] "Downloading provider manifests" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capa-system/aws" namespace="capa-system" name="aws" reconcileID="cfbcc867-7dfc-494a-9fce-cb55edd927d1"
I0328 10:32:50.485596       1 phases.go:182] "Loading provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capa-system/aws" namespace="capa-system" name="aws" reconcileID="cfbcc867-7dfc-494a-9fce-cb55edd927d1"
I0328 10:32:50.686709       1 phases.go:448] "Fetching provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capa-system/aws" namespace="capa-system" name="aws" reconcileID="cfbcc867-7dfc-494a-9fce-cb55edd927d1"
I0328 10:32:53.567809       1 phases.go:508] "Version changes detected, updating existing components" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capa-system/aws" namespace="capa-system" name="aws" reconcileID="cfbcc867-7dfc-494a-9fce-cb55edd927d1"
I0328 10:32:58.773091       1 reflector.go:368] Caches populated for *v1.CustomResourceDefinition from pkg/mod/k8s.io/client-go@v0.31.5/tools/cache/reflector.go:243
I0328 10:32:59.073188       1 warning_handler.go:65] "spec.template.spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution[1].preference.matchExpressions[0].key: node-role.kubernetes.io/master is use \"node-role.kubernetes.io/control-plane\" instead" logger="KubeAPIWarningLogger"
I0328 10:32:59.073235       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capa-system/capa-controller-manager" namespace="capa-system" name="capa-controller-manager" reconcileID="1231d462-d828-4dd6-bacd-49b0e7e24056"
I0328 10:32:59.085417       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capa-system/capa-controller-manager" namespace="capa-system" name="capa-controller-manager" reconcileID="4ac0064f-5a8f-4854-8f6c-24ca07a2565a"
I0328 10:32:59.096922       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capa-system/capa-controller-manager" namespace="capa-system" name="capa-controller-manager" reconcileID="fca7ca3a-dc15-45a4-8120-cd44e0f4f6ae"
I0328 10:33:02.533111       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capa-system/capa-controller-manager" namespace="capa-system" name="capa-controller-manager" reconcileID="9574d994-e296-4812-89d8-c338b440c264"
I0328 10:33:02.544860       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capa-system/capa-controller-manager" namespace="capa-system" name="capa-controller-manager" reconcileID="a62fbb27-6e45-432e-a01e-c716c9c07cd0"
I0328 10:33:02.550790       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capa-system/capa-controller-manager" namespace="capa-system" name="capa-controller-manager" reconcileID="e1813283-8526-46b4-8da6-465f031d2f7a"
I0328 10:33:02.592973       1 phases.go:517] "Provider successfully upgraded" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capa-system/aws" namespace="capa-system" name="aws" reconcileID="cfbcc867-7dfc-494a-9fce-cb55edd927d1"
I0328 10:33:02.690555       1 genericprovider_controller.go:92] "Reconciling provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capd-system/docker" namespace="capd-system" name="docker" reconcileID="39858612-cd46-4b2c-9089-4fe18d5bdb89"
I0328 10:33:02.691707       1 preflight_checks.go:58] "Performing preflight checks" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capd-system/docker" namespace="capd-system" name="docker" reconcileID="39858612-cd46-4b2c-9089-4fe18d5bdb89"
I0328 10:33:02.693456       1 preflight_checks.go:199] "Preflight checks passed" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capd-system/docker" namespace="capd-system" name="docker" reconcileID="39858612-cd46-4b2c-9089-4fe18d5bdb89"
I0328 10:33:03.567811       1 phases.go:182] "Loading provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capd-system/docker" namespace="capd-system" name="docker" reconcileID="39858612-cd46-4b2c-9089-4fe18d5bdb89"
I0328 10:33:03.585233       1 phases.go:448] "Fetching provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capd-system/docker" namespace="capd-system" name="docker" reconcileID="39858612-cd46-4b2c-9089-4fe18d5bdb89"
I0328 10:33:03.988896       1 phases.go:534] "Installing provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capd-system/docker" namespace="capd-system" name="docker" reconcileID="39858612-cd46-4b2c-9089-4fe18d5bdb89"
I0328 10:33:04.074329       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capa-system/capa-controller-manager" namespace="capa-system" name="capa-controller-manager" reconcileID="788b68f7-b784-4487-9680-aeabee5bf632"
I0328 10:33:04.392651       1 phases.go:545] "Provider successfully installed" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capd-system/docker" namespace="capd-system" name="docker" reconcileID="39858612-cd46-4b2c-9089-4fe18d5bdb89"
I0328 10:33:04.401983       1 genericprovider_controller.go:92] "Reconciling provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capg-system/gcp" namespace="capg-system" name="gcp" reconcileID="ab371cd7-dc8d-4dbd-9901-97ff3018e8e2"
I0328 10:33:04.466962       1 preflight_checks.go:58] "Performing preflight checks" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capg-system/gcp" namespace="capg-system" name="gcp" reconcileID="ab371cd7-dc8d-4dbd-9901-97ff3018e8e2"
I0328 10:33:04.469118       1 preflight_checks.go:199] "Preflight checks passed" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capg-system/gcp" namespace="capg-system" name="gcp" reconcileID="ab371cd7-dc8d-4dbd-9901-97ff3018e8e2"
I0328 10:33:04.771678       1 phases.go:182] "Loading provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capg-system/gcp" namespace="capg-system" name="gcp" reconcileID="ab371cd7-dc8d-4dbd-9901-97ff3018e8e2"
I0328 10:33:05.267399       1 phases.go:448] "Fetching provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capg-system/gcp" namespace="capg-system" name="gcp" reconcileID="ab371cd7-dc8d-4dbd-9901-97ff3018e8e2"
I0328 10:33:05.767537       1 phases.go:508] "Version changes detected, updating existing components" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capg-system/gcp" namespace="capg-system" name="gcp" reconcileID="ab371cd7-dc8d-4dbd-9901-97ff3018e8e2"

[furkatgofurov7]

@cpinjani can we have capi operator pod definition or anything that could give some more hints why it is OOM killed?

One more place it could be recorded might be: kubectl events --for pod/<capi operator pod>

[cpinjani]

Adding more pod details.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2025-03-28T10:31:00Z"
  generateName: rancher-turtles-cluster-api-operator-59b5f8d9db-
  labels:
    app: cluster-api-operator
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: rancher-turtles
    app.kubernetes.io/name: cluster-api-operator
    clusterctl.cluster.x-k8s.io/core: capi-operator
    control-plane: controller-manager
    pod-template-hash: 59b5f8d9db
  name: rancher-turtles-cluster-api-operator-59b5f8d9db-59jj2
  namespace: rancher-turtles-system
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: rancher-turtles-cluster-api-operator-59b5f8d9db
    uid: 720b98a5-0f13-4f25-b318-ed383d9f99df
  resourceVersion: "18026"
  uid: e1002c2a-3c00-4833-84fe-62e82d841343
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values:
            - amd64
            - arm64
            - ppc64le
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
  automountServiceAccountToken: true
  containers:
  - args:
    - --v=2
    - --health-addr=:9440
    - --diagnostics-address=:8443
    - --leader-elect=true
    command:
    - /manager
    image: registry.rancher.com/rancher/cluster-api-operator:v0.17.1
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz
        port: 9440
        scheme: HTTP
      initialDelaySeconds: 15
      periodSeconds: 20
      successThreshold: 1
      timeoutSeconds: 1
    name: manager
    ports:
    - containerPort: 9443
      name: webhook-server
      protocol: TCP
    - containerPort: 8443
      name: metrics
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /readyz
        port: 9440
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: 100m
        memory: 150Mi
      requests:
        cpu: 100m
        memory: 100Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
    volumeMounts:
    - mountPath: /tmp/k8s-webhook-server/serving-certs
      name: cert
      readOnly: true
    - mountPath: /config
      name: clusterctl-config
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-l9z7k
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: cpinjani-test1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: capi-operator-manager
  serviceAccountName: capi-operator-manager
  terminationGracePeriodSeconds: 10
  tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: cert
    secret:
      defaultMode: 420
      secretName: capi-operator-webhook-service-cert
  - configMap:
      defaultMode: 420
      name: clusterctl-config
    name: clusterctl-config
  - name: kube-api-access-l9z7k
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2025-03-28T10:31:05Z"
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
    lastTransitionTime: "2025-03-28T10:31:00Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2025-03-28T10:37:10Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2025-03-28T10:37:10Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2025-03-28T10:31:00Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://51fefc4450231a17542c7af74567f9cdea47de26d64c09a9800a264aea852f19
    image: registry.rancher.com/rancher/cluster-api-operator:v0.17.1
    imageID: registry.rancher.com/rancher/cluster-api-operator@sha256:12dac79159467f46718155a1e540912236ba984726e2ed4c1eb77d8c728ae493
    lastState:
      terminated:
        containerID: containerd://64daac610696c6e9fa1b3826ab7d8a4502674570b4b70ce2619a83b8b29ab6e6
        exitCode: 137
        finishedAt: "2025-03-28T10:36:10Z"
        message: |
          er="capi-kubeadm-control-plane-system/kubeadm" namespace="capi-kubeadm-control-plane-system" name="kubeadm" reconcileID="4f646d37-1f50-4028-867a-41210a50890d"
          I0328 10:35:46.215735       1 genericprovider_controller.go:92] "Reconciling provider" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="629e2ed3-b7cb-4e84-bca1-9997176dfcbc"
          I0328 10:35:46.216922       1 genericprovider_controller.go:141] "No changes detected, skipping further steps" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="rke2-control-plane-system/rke2" namespace="rke2-control-plane-system" name="rke2" reconcileID="629e2ed3-b7cb-4e84-bca1-9997176dfcbc"
          I0328 10:35:46.828247       1 phases.go:182] "Loading provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="a43e146e-4a10-4b5a-8b12-45b3083034c4"
          I0328 10:35:47.414230       1 phases.go:448] "Fetching provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="a43e146e-4a10-4b5a-8b12-45b3083034c4"
          I0328 10:36:02.215742       1 phases.go:508] "Version changes detected, updating existing components" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="a43e146e-4a10-4b5a-8b12-45b3083034c4"
          I0328 10:36:07.416161       1 reflector.go:368] Caches populated for *v1.CustomResourceDefinition from pkg/mod/k8s.io/client-go@v0.31.5/tools/cache/reflector.go:243
        reason: OOMKilled
        startedAt: "2025-03-28T10:35:29Z"
    name: manager
    ready: true
    restartCount: 4
    started: true
    state:
      running:
        startedAt: "2025-03-28T10:36:56Z"
    volumeMounts:
    - mountPath: /tmp/k8s-webhook-server/serving-certs
      name: cert
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /config
      name: clusterctl-config
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-l9z7k
      readOnly: true
      recursiveReadOnly: Disabled
  hostIP: 10.128.0.71
  hostIPs:
  - ip: 10.128.0.71
  phase: Running
  podIP: 10.42.0.62
  podIPs:
  - ip: 10.42.0.62
  qosClass: Burstable
  startTime: "2025-03-28T10:31:00Z"

Name:             rancher-turtles-cluster-api-operator-59b5f8d9db-59jj2
Namespace:        rancher-turtles-system
Priority:         0
Service Account:  capi-operator-manager
Node:             cpinjani-test1/10.128.0.71
Start Time:       Fri, 28 Mar 2025 10:31:00 +0000
Labels:           app=cluster-api-operator
                  app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=rancher-turtles
                  app.kubernetes.io/name=cluster-api-operator
                  clusterctl.cluster.x-k8s.io/core=capi-operator
                  control-plane=controller-manager
                  pod-template-hash=59b5f8d9db
Annotations:      <none>
Status:           Running
IP:               10.42.0.62
IPs:
  IP:           10.42.0.62
Controlled By:  ReplicaSet/rancher-turtles-cluster-api-operator-59b5f8d9db
Containers:
  manager:
    Container ID:  containerd://64daac610696c6e9fa1b3826ab7d8a4502674570b4b70ce2619a83b8b29ab6e6
    Image:         registry.rancher.com/rancher/cluster-api-operator:v0.17.1
    Image ID:      registry.rancher.com/rancher/cluster-api-operator@sha256:12dac79159467f46718155a1e540912236ba984726e2ed4c1eb77d8c728ae493
    Ports:         9443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /manager
    Args:
      --v=2
      --health-addr=:9440
      --diagnostics-address=:8443
      --leader-elect=true
    State:       Running
      Started:   Fri, 28 Mar 2025 10:35:29 +0000
    Last State:  Terminated
      Reason:    OOMKilled
      Message:   ntroller-manager" reconcileID="68a2282e-2b18-4c8d-bea6-4c22844951e4"
I0328 10:34:53.559134       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capz-system/azureserviceoperator-controller-manager" namespace="capz-system" name="azureserviceoperator-controller-manager" reconcileID="b1d2208d-eb3d-40c6-b976-6d860fadfbee"
I0328 10:34:53.579352       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capz-system/azureserviceoperator-controller-manager" namespace="capz-system" name="azureserviceoperator-controller-manager" reconcileID="34fde834-1833-489e-b5ba-1ba5a3123805"
I0328 10:34:54.615526       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capz-system/capz-controller-manager" namespace="capz-system" name="capz-controller-manager" reconcileID="4f547150-6993-4487-a025-cd89bb37e20a"
I0328 10:34:54.628659       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capz-system/capz-controller-manager" namespace="capz-system" name="capz-controller-manager" reconcileID="ef8519a5-8c1d-44d3-b488-9ea9b2328747"
I0328 10:34:54.645724       1 healthcheck_controller.go:128] "Checking provider health" controller="healthcheck-operator.cluster.x-k8s.io/v1alpha2, Kind=InfrastructureProvider" controllerGroup="apps" controllerKind="Deployment" Deployment="capz-system/capz-controller-manager" namespace="capz-system" name="capz-controller-manager" reconcileID="050fcd0e-1180-411e-838d-19204933f342"

      Exit Code:    137
      Started:      Fri, 28 Mar 2025 10:34:14 +0000
      Finished:     Fri, 28 Mar 2025 10:34:57 +0000
    Ready:          True
    Restart Count:  3
    Limits:
      cpu:     100m
      memory:  150Mi
    Requests:
      cpu:        100m
      memory:     100Mi
    Liveness:     http-get http://:9440/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:    http-get http://:9440/readyz delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /config from clusterctl-config (ro)
      /tmp/k8s-webhook-server/serving-certs from cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-l9z7k (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  capi-operator-webhook-service-cert
    Optional:    false
  clusterctl-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      clusterctl-config
    Optional:  false
  kube-api-access-l9z7k:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node-role.kubernetes.io/control-plane:NoSchedule
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  4m52s                default-scheduler  Successfully assigned rancher-turtles-system/rancher-turtles-cluster-api-operator-59b5f8d9db-59jj2 to cpinjani-test1
  Normal   Pulling    4m52s                kubelet            Pulling image "registry.rancher.com/rancher/cluster-api-operator:v0.17.1"
  Normal   Pulled     4m47s                kubelet            Successfully pulled image "registry.rancher.com/rancher/cluster-api-operator:v0.17.1" in 4.605s (4.605s including waiting). Image size: 39487816 bytes.
  Warning  BackOff    37s (x7 over 117s)   kubelet            Back-off restarting failed container manager in pod rancher-turtles-cluster-api-operator-59b5f8d9db-59jj2_rancher-turtles-system(e1002c2a-3c00-4833-84fe-62e82d841343)
  Normal   Created    23s (x4 over 4m47s)  kubelet            Created container manager
  Normal   Started    23s (x4 over 4m47s)  kubelet            Started container manager
  Normal   Pulled     23s (x3 over 2m37s)  kubelet            Container image "registry.rancher.com/rancher/cluster-api-operator:v0.17.1" already present on machine

[Danil-Grigorev]

@furkatgofurov7 Operator has major issues with memory usage and repetitive cluster wide lists of resources on each reconcile. It is constantly near the limit of the requested resource quotas in both CPU and memory, even when a minimal set of providers is installed from what I’ve seen.

[cpinjani]

Among the cloud providers, azure infrastructure provider seem to be affected. Otherwise the upgrade worked fine for other ones. Also it is not occurring on fresh install of v0.18.0-rc.1

Logs:

I0328 16:07:32.398683       1 preflight_checks.go:199] "Preflight checks passed" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="740381d8-97db-44b2-8621-cd65c87a3f76"
I0328 16:07:33.000762       1 phases.go:182] "Loading provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="740381d8-97db-44b2-8621-cd65c87a3f76"
I0328 16:07:33.695343       1 phases.go:448] "Fetching provider" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="740381d8-97db-44b2-8621-cd65c87a3f76"
I0328 16:07:48.602086       1 phases.go:508] "Version changes detected, updating existing components" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/azure" namespace="capz-system" name="azure" reconcileID="740381d8-97db-44b2-8621-cd65c87a3f76"
I0328 16:07:52.795045       1 reflector.go:368] Caches populated for *v1.CustomResourceDefinition from pkg/mod/k8s.io/client-go@v0.31.5/tools/cache/reflector.go:243

Fresh install:

Providers versions after upgrade:

[furkatgofurov7]

@cpinjani is issue with azure provider not being upgraded, because in the first screenshot I see it on version v1.19.0 and that is the version on v0.18.0-rc1. We bumped it to v1.19.1 on v0.18.0 release only.

[cpinjani]

@furkatgofurov7 I checked the issue reproducible after upgrade to turtles v0.18.0 (capz - v1.19.1)