Somehow stuck on outdated Kubernetes version, talosconfig apply or talosctl upgrade-k8s refuse to update

[kastl-ars]

Dear all,

I think I somehow reverted back to an older k8s version when I implemented the talos-backup recently.

So now I seem to be stuck with Talos 1.13.2 and k8s 1.30.3. And both talosconfig apply ... and talosctl upgrade-k8s --to 1.31.0 refuse to do anything (see below).

Any idea how to safely break the cycle? Revert back to an older Talos release?

Manual k8s upgrade](https://docs.siderolabs.com/kubernetes-guides/advanced-guides/upgrading-kubernetes#manual-kubernetes-upgrade)?

Kind Regards,
Johannes

$ talosctl --talosconfig ./talosconfig upgrade-k8s --to 1.31.0
automatically detected the lowest Kubernetes version 1.30.3
discovered controlplane nodes ["XXX"]
discovered worker nodes []
> "XXX": Talos version 1.13.2 is compatible with Kubernetes version 1.31.0
checking for removed Kubernetes component flags
checking for removed Kubernetes API resource versions
 > "192.168.99.241": pre-pulling registry.k8s.io/kube-apiserver:v1.31.0
 > "192.168.99.241": pre-pulling registry.k8s.io/kube-controller-manager:v1.31.0
 > "192.168.99.241": pre-pulling registry.k8s.io/kube-scheduler:v1.31.0
 > "192.168.99.241": pre-pulling ghcr.io/siderolabs/kubelet:v1.31.0
updating "kube-apiserver" to version "1.31.0"
 > "XXX": starting update
 > update kube-apiserver: v1.30.3 -> 1.31.0
failed updating service "kube-apiserver": error updating node "192.168.99.241": error patching node config: error applying config: rpc error: code = InvalidArgument desc = 1 error occurred:
        * v1alpha1.Config: 3 errors occurred:
        * kubelet image is not valid: version of Kubernetes 1.30.3 is too old to be used with Talos 1.13.2
        * kube-controller-manager image is not valid: version of Kubernetes 1.30.3 is too old to be used with Talos 1.13.2
        * kube-scheduler image is not valid: version of Kubernetes 1.30.3 is too old to be used with Talos 1.13.2
$

$ talosctl apply-config --talosconfig=./talosconfig --mode reboot --file controlplane.yaml
error applying new configuration: 1 error occurred:
        * XXX: rpc error: code = InvalidArgument desc = 1 error occurred:
        * v1alpha1.Config: 3 errors occurred:
        * kube-apiserver image is not valid: version of Kubernetes 1.30.3 is too old to be used with Talos 1.13.2
        * kube-controller-manager image is not valid: version of Kubernetes 1.30.3 is too old to be used with Talos 1.13.2
        * kube-scheduler image is not valid: version of Kubernetes 1.30.3 is too old to be used with Talos 1.13.2

$

[frezbo]

Please update talosctl to the latest

[kastl-ars]

Thanks for the fast reply, it already is at 1.13.2:

Client:
        Tag:         v1.13.2
        SHA:         c5d7c653
        Built:       2026-05-12T13:35:25Z
        Go version:  go1.26.3
        OS/Arch:     linux/amd64

[smira]

I wonder how you ended up like this (?) - Talos shouldn't let you boot even with unsupported Kubernetes version in the machine config.

[kastl-ars]

I fell into the "Do not store complete machineconfigs" trap, I guess.

My current guess what happened:

• I applied the machine configuration to enable the talos-backup and downgraded to 1.30.3 in the process
• I upgraded Talos from 1.12.6 to 1.12.7 and then to 1.13.0 and 1.13.2 (without warnings or issues)
• I noticed the k8s was largely outdated and tried to upgraded => this discussion... :-)

[kastl-ars]

For the record: Using an older talosctl (1.12.7) does not work.

[smira]

yes, we have a bug with validation, it should have prevented you from even booting/upgrading.

Right now the only option is to manually patch all references to Kubernetes image on each machine to a supported version (all at once).

[kastl-ars]

OK, I'll try that. Is there a list of k8s version supported by each Talos release? (I did search, but might have missed it?)

[smira]

https://docs.siderolabs.com/talos/v1.13/getting-started/support-matrix

[kastl-ars]

Hmm, I had a look but am not sure what exactly I would need to patch. Is it enough to patch the Pods for the api server, controller-manager and scheduler? Or will those be overwritten as they are generated by Talos?

In the latter case, what exactly would I need to patch? As I found no deployments, statefulsets or other custom resource (at least in the Kubernetes API)

[frezbo]

patching the machine config for the kubernetes versions for apiserver, scheduler kubelet and kube-proxy

[kastl-ars]

Patching using talosctl ... patch mc ... does also not work... :-(