Webhook error: [FATAL] rancher-start::authServer leader func failed be executed... #1130
Description
Hello,
We are running Rancher in a kubernetes cluster and since we updated last week from v2.12.1 to v2.12.2 we are continuously experience webhook error when rancher toolchain is restarted.
The fixes from here rancher/rancher#42611 holds until next restart.
Here is the complete startup log with the failing error:
Restoring git repositories:
- /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721/.git
Your branch is up to date with 'origin/release-v2.12'.
/var/lib/rancher
- /var/lib/rancher-data/local-catalogs/v2/rancher-rke2-charts/675f1b63a0a83905972dcab2794479ed599a6f41b86cd6193d69472d0fa889c9/.git
Your branch is up to date with 'origin/main'.
/var/lib/rancher
- /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974/.git
Your branch is up to date with 'origin/main'.
/var/lib/rancher
INFO: Running k3s server --cluster-init --cluster-reset
2025/10/07 10:54:41 [INFO] Rancher version v2.12.2 (52946dcf82e245b2a133d437740300b8d35e69ca) is starting
2025/10/07 10:54:41 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:embedded Debug:false Trace:false NoCACerts:true AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:1 AuditLogMaxsize:100 AuditLogMaxbackup:1 AuditLogLevel:1 AuditLogEnabled:true Features: ClusterRegistry: AggregationRegistrationTimeout:5m0s}
2025/10/07 10:54:41 [INFO] Listening on /tmp/log.sock
2025/10/07 10:54:41 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
2025/10/07 10:54:43 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
2025/10/07 10:54:45 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
2025/10/07 10:54:47 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
2025/10/07 10:54:49 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
2025/10/07 10:54:51 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
2025/10/07 10:54:53 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
2025/10/07 10:54:55 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
2025/10/07 10:54:57 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
2025/10/07 10:54:59 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
2025/10/07 10:55:01 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
2025/10/07 10:55:04 [INFO] Running in single server mode, will not peer connections
2025/10/07 10:55:04 [INFO] Scanning NodeTemplates in namespace: cattle-global-nt, group: nodetemplates.management.cattle.io
2025/10/07 10:55:04 [INFO] Scanning ClusterTemplates in namespace: cattle-global-data, group: clustertemplates.management.cattle.io
I1007 10:55:04.089364 433 warnings.go:110] "Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice"
2025/10/07 10:55:04 [INFO] Applying CRD features.management.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD auditpolicies.auditlog.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD clusterrepos.catalog.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD clusters.provisioning.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD custommachines.rke.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD etcdsnapshots.rke.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD rkebootstraps.rke.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD rkebootstraptemplates.rke.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD rkeclusters.rke.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD rkecontrolplanes.rke.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD clusterroletemplatebindings.management.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD dynamicschemas.management.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD globalroles.management.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD globalrolebindings.management.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD nodedrivers.management.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD projects.management.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD projectroletemplatebindings.management.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD roletemplates.management.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD clusterproxyconfigs.management.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD uiplugins.catalog.cattle.io
2025/10/07 10:55:05 [INFO] Updating embedded CRD secretrequests.telemetry.cattle.io
2025/10/07 10:55:05 [INFO] Applying CRD plans.upgrade.cattle.io
2025/10/07 10:55:05 [INFO] Applying CRD navlinks.ui.cattle.io
2025/10/07 10:55:06 [INFO] Applying CRD podsecurityadmissionconfigurationtemplates.management.cattle.io
2025/10/07 10:55:06 [INFO] Applying CRD clusters.management.cattle.io
2025/10/07 10:55:06 [INFO] Applying CRD apiservices.management.cattle.io
2025/10/07 10:55:06 [INFO] Applying CRD clusterregistrationtokens.management.cattle.io
2025/10/07 10:55:06 [INFO] Applying CRD settings.management.cattle.io
2025/10/07 10:55:06 [INFO] Applying CRD preferences.management.cattle.io
2025/10/07 10:55:06 [INFO] Applying CRD features.management.cattle.io
2025/10/07 10:55:06 [INFO] Applying CRD operations.catalog.cattle.io
2025/10/07 10:55:06 [INFO] Applying CRD apps.catalog.cattle.io
2025/10/07 10:55:06 [INFO] Applying CRD fleetworkspaces.management.cattle.io
2025/10/07 10:55:06 [INFO] Applying CRD managedcharts.management.cattle.io
2025/10/07 10:55:06 [INFO] Applying CRD clusters.cluster.x-k8s.io
2025/10/07 10:55:06 [INFO] Applying CRD machinedeployments.cluster.x-k8s.io
2025/10/07 10:55:06 [INFO] Applying CRD machinedrainrules.cluster.x-k8s.io
2025/10/07 10:55:06 [INFO] Applying CRD machinehealthchecks.cluster.x-k8s.io
2025/10/07 10:55:06 [INFO] Applying CRD machines.cluster.x-k8s.io
2025/10/07 10:55:06 [INFO] Applying CRD machinesets.cluster.x-k8s.io
2025/10/07 10:55:06 [INFO] Configuring auth server API body limit to 1048576 bytes
2025/10/07 10:55:06 [INFO] creating imperative extension apiserver resources
2025/10/07 10:55:06 [INFO] RDPClient: Checking if dialer is built...
2025/10/07 10:55:06 [INFO] RDPClient: Dialer is not built yet, waiting 5 secs to re-check.
2025/10/07 10:55:06 [INFO] starting imperative api cert rotator
2025/10/07 10:55:06 [INFO] imperative api APIService cert updated
2025/10/07 10:55:07 [INFO] Successfully installed useractivity store
2025/10/07 10:55:07 [INFO] Successfully installed token store
2025/10/07 10:55:07 [INFO] Successfully installed kubeconfig store
I1007 10:55:07.489807 433 handler.go:288] Adding GroupVersion ext.cattle.io v1 to ResourceManager
I1007 10:55:08.595385 433 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1007 10:55:08.595415 433 shared_informer.go:350] "Waiting for caches to sync" controller="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1007 10:55:08.595355 433 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController
I1007 10:55:08.595443 433 shared_informer.go:350] "Waiting for caches to sync" controller="RequestHeaderAuthRequestController"
I1007 10:55:08.595733 433 secure_serving.go:211] Serving securely on [::]:6666
I1007 10:55:08.595827 433 tlsconfig.go:243] "Starting DynamicServingCertificateController"
I1007 10:55:08.695861 433 shared_informer.go:357] "Caches are synced" controller="RequestHeaderAuthRequestController"
I1007 10:55:08.695872 433 shared_informer.go:357] "Caches are synced" controller="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
2025/10/07 10:55:10 [INFO] Configuring public API body limit to 1048576 bytes
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=APIService controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=KontainerDriver controller
2025/10/07 10:55:10 [INFO] Starting /v1, Kind=Secret controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=Node controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=GlobalRole controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=ClusterRoleTemplateBinding controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=ClusterProxyConfig controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=DynamicSchema controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=AuthConfig controller
2025/10/07 10:55:10 [INFO] RDPClient: certificate updated successfully
2025/10/07 10:55:10 [INFO] Starting apiextensions.k8s.io/v1, Kind=CustomResourceDefinition controller
2025/10/07 10:55:10 [INFO] Starting cluster.x-k8s.io/v1beta1, Kind=Machine controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=Feature controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=ProjectRoleTemplateBinding controller
2025/10/07 10:55:10 [INFO] Starting /v1, Kind=Namespace controller
2025/10/07 10:55:10 [INFO] Starting provisioning.cattle.io/v1, Kind=Cluster controller
2025/10/07 10:55:10 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2025/10/07 10:55:10 [INFO] Starting API controllers
2025/10/07 10:55:10 [INFO] Starting /v1, Kind=ServiceAccount controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2025/10/07 10:55:10 [INFO] Starting catalog.cattle.io/v1, Kind=ClusterRepo controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=Preference controller
2025/10/07 10:55:10 [INFO] Starting catalog.cattle.io/v1, Kind=UIPlugin controller
2025/10/07 10:55:10 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=ClusterRegistrationToken controller
I1007 10:55:10.933924 433 leaderelection.go:257] attempting to acquire leader lease kube-system/cattle-controllers...
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=User controller
2025/10/07 10:55:10 [INFO] Starting rke.cattle.io/v1, Kind=RKEBootstrap controller
2025/10/07 10:55:10 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2025/10/07 10:55:10 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2025/10/07 10:55:10 [INFO] Starting rke.cattle.io/v1, Kind=ETCDSnapshot controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=Setting controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=Project controller
2025/10/07 10:55:10 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=RoleTemplate controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2025/10/07 10:55:10 [INFO] Starting management.cattle.io/v3, Kind=NodeDriver controller
2025/10/07 10:55:10 [INFO] getReleaseConfig: reloading config for rke2
2025/10/07 10:55:10 [INFO] getReleaseConfig: reloading config for k3s
2025/10/07 10:55:10 [INFO] Starting /v1, Kind=ConfigMap controller
2025/10/07 10:55:10 [INFO] Starting auditlog.cattle.io/v1, Kind=AuditPolicy controller
I1007 10:55:10.941318 433 leaderelection.go:271] successfully acquired lease kube-system/cattle-controllers
2025/10/07 10:55:10 [INFO] wrangler-newContext leader func executed successfully
2025/10/07 10:55:10 [INFO] nodedriver-register leader func executed successfully
2025/10/07 10:55:10 [INFO] Reconciling GlobalRoles
2025/10/07 10:55:10 [INFO] Starting ServiceAccountSecretCleaner with 0 secrets
2025/10/07 10:55:10 [INFO] rancher-new leader func executed successfully
2025/10/07 10:55:10 [INFO] Starting cluster controllers for local
2025/10/07 10:55:10 [INFO] Reconciling RoleTemplates
2025/10/07 10:55:11 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
2025/10/07 10:55:11 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2025/10/07 10:55:11 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2025/10/07 10:55:11 [INFO] Starting management.cattle.io/v3, Kind=User controller
2025/10/07 10:55:11 [INFO] waiting for telemetry manager to start...
2025/10/07 10:55:11 [INFO] Starting /v1, Kind=Secret controller
2025/10/07 10:55:11 [INFO] Starting cluster.x-k8s.io/v1beta1, Kind=Cluster controller
2025/10/07 10:55:11 [INFO] initializing required info for telemetry manager...
2025/10/07 10:55:11 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2025/10/07 10:55:11 [INFO] Starting rke.cattle.io/v1, Kind=RKEControlPlane controller
2025/10/07 10:55:11 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2025/10/07 10:55:11 [INFO] initialized required info for telemetry manager
2025/10/07 10:55:11 [INFO] telemetry manager info not available yet, re-queing check...
2025/10/07 10:55:11 [INFO] starting telemetry gathering
2025/10/07 10:55:11 [INFO] telemetry manager started
2025/10/07 10:55:11 [FATAL] rancher-start::authServer leader func failed be executed: failed to add authconfig data: Internal error occurred: failed calling webhook "rancher.cattle.io.authconfigs.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/authconfigs.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
After we "temporarily" fix the webhook error we also noticed these logs
025/10/07 08:02:18 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.12/data.json in [0x10baa820]
2025/10/07 08:02:18 [INFO] getReleaseConfig: reloading config for rke2
2025/10/07 08:02:18 [INFO] Selected pod "api-extension-944888f69-dt7qh" for label "app=api-extension"
2025/10/07 08:02:18 [ERROR] Non-restartable error: error upgrading connection: error dialing backend: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, 10.42.12.45, not 10.42.12.18
2025/10/07 08:02:18 [ERROR] RDPClient: error upgrading connection: error dialing backend: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, 10.42.12.45, not 10.42.12.18
2025/10/07 08:02:18 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.12/data.json in [0x10baa820]
2025/10/07 08:02:18 [INFO] getReleaseConfig: reloading config for k3s
2025/10/07 08:02:18 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.12/data.json in [0x10baa820]
2025/10/07 08:02:18 [INFO] getReleaseConfig: reloading config for rke2
I1007 08:02:19.007967 439 warnings.go:110] "Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice"
I1007 08:02:19.114318 439 warnings.go:110] "Warning: v1 ComponentStatus is deprecated in v1.19+"
and these logs
2025/10/07 08:02:19 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.12/data.json in [0x10baa820]
2025/10/07 08:02:19 [INFO] getReleaseConfig: reloading config for k3s
2025/10/07 08:02:19 [INFO] CacheFor STARTS creating informer for /v1, Kind=Namespace
2025/10/07 08:02:19 [INFO] Started SQL cache garbage collection for _v1_Namespace (interval=15m0s, keep=1000)
2025/10/07 08:02:19 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.12/data.json in [0x10baa820]
2025/10/07 08:02:19 [INFO] getReleaseConfig: reloading config for rke2
2025/10/07 08:02:19 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 7, ObservedGeneration is 7, LastUpdated plus interval is 2025-10-07 09:02:19 +0000 UTC, now is 2025-10-07 08:02:19.777209316 +0000 UTC
2025/10/07 08:02:19 [INFO] CacheFor IS DONE creating informer for /v1, Kind=Namespace (took 126.265471ms)
2025/10/07 08:02:19 [ERROR] Failed to read API for groups map[ext.cattle.io/v1:stale GroupVersion discovery: ext.cattle.io/v1]
2025/10/07 08:02:20 [INFO] Skipping handler for clusterrepo rancher-rke2-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-10-07 09:02:19 +0000 UTC, now is 2025-10-07 08:02:20.051150984 +0000 UTC
2025/10/07 08:02:20 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.12/data.json in [0x10baa820]
I1007 08:02:20.286404 439 warnings.go:110] "Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice"
I1007 08:02:20.295392 439 warnings.go:110] "Warning: v1 ComponentStatus is deprecated in v1.19+"
2025/10/07 08:02:20 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.12/data.json in [0x10baa820]
2025/10/07 08:02:20 [INFO] Selected pod "api-extension-944888f69-dt7qh" for label "app=api-extension"
2025/10/07 08:02:20 [ERROR] Non-restartable error: error upgrading connection: error dialing backend: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, 10.42.12.45, not 10.42.12.18
2025/10/07 08:02:20 [ERROR] RDPClient: error upgrading connection: error dialing backend: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, 10.42.12.45, not 10.42.12.18
2025/10/07 08:02:21 [INFO] Selected pod "api-extension-944888f69-dt7qh" for label "app=api-extension"
2025/10/07 08:02:21 [ERROR] Non-restartable error: error upgrading connection: error dialing backend: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, 10.42.12.45, not 10.42.12.18
2025/10/07 08:02:21 [ERROR] RDPClient: error upgrading connection: error dialing backend: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, 10.42.12.45, not 10.42.12.18
2025/10/07 08:02:21 [INFO] ServiceAccountSecretCleaner has no secrets remaining - terminating at 5.000229865s
2025/10/07 08:02:22 [INFO] Selected pod "api-extension-944888f69-dt7qh" for label "app=api-extension"
2025/10/07 08:02:22 [ERROR] Non-restartable error: error upgrading connection: error dialing backend: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, 10.42.12.45, not 10.42.12.18
2025/10/07 08:02:22 [ERROR] RDPClient: error upgrading connection: error dialing backend: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, 10.42.12.45, not 10.42.12.18
2025/10/07 08:02:23 [INFO] Selected pod "api-extension-944888f69-dt7qh" for label "app=api-extension"
2025/10/07 08:02:23 [ERROR] Non-restartable error: error upgrading connection: error dialing backend: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, 10.42.12.45, not 10.42.12.18
2025/10/07 08:02:23 [ERROR] RDPClient: error upgrading connection: error dialing backend: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, 10.42.12.45, not 10.42.12.18
2025/10/07 08:02:23 [INFO] Skipping handler for clusterrepo rancher-partner-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-10-07 09:02:23 +0000 UTC, now is 2025-10-07 08:02:23.898357503 +0000 UTC
2025/10/07 08:02:24 [INFO] Selected pod "api-extension-944888f69-dt7qh" for label "app=api-extension"
2025/10/07 08:02:24 [INFO] RDPClient: connecting to wss://127.0.0.1:5555/connect
2025/10/07 08:02:24 [INFO] Connecting to proxy
2025/10/07 08:02:24 [INFO] Connected to proxy
2025/10/07 08:02:24 [INFO] RDPClient: remotedialer session connected!
2025/10/07 08:02:25 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 7, ObservedGeneration is 7, LastUpdated plus interval is 2025-10-07 09:02:19 +0000 UTC, now is 2025-10-07 08:02:25.777263111 +0000 UTC
I1007 08:02:26.513268 439 warnings.go:110] "Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice"
I1007 08:02:26.559215 439 warnings.go:110] "Warning: v1 ComponentStatus is deprecated in v1.19+"
2025/10/07 08:02:26 [INFO] Watching metadata for ext.cattle.io/v1, Kind=Kubeconfig
2025/10/07 08:02:26 [INFO] Watching metadata for ext.cattle.io/v1, Kind=Token
2025/10/07 08:02:29 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 7, ObservedGeneration is 7, LastUpdated plus interval is 2025-10-07 09:02:19 +0000 UTC, now is 2025-10-07 08:02:29.94275566 +0000 UTC
2025/10/07 08:02:31 [ERROR] [updateClusterHealth] Failed to update cluster [local]: Internal error occurred: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2025/10/07 08:02:52 [INFO] [systemcharts] found deployment cattle-system/system-upgrade-controller with index of target finalzier = 0
I1007 08:02:54.590446 439 warnings.go:110] "Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice"
I1007 08:02:54.613349 439 warnings.go:110] "Warning: v1 ComponentStatus is deprecated in v1.19+"
2025/10/07 08:03:04 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 7, ObservedGeneration is 7, LastUpdated plus interval is 2025-10-07 09:02:19 +0000 UTC, now is 2025-10-07 08:03:04.27298165 +0000 UTC
What can be the reason of all of these?
Best Regards,
George