Prevent Talos and Kubernetes upgrades from happening at the same time

[onedr0p]

I might be having a similar issue (except I'm not seeing any error related to tls certificate).
I can open a separate issue if this is unrelated. If you need any testing or additional logs, please let me know.

I also just recreated the TalosUpgrade resource, and it proceeds normally. link to my config

2026-01-07T15:23:46.164393387Z 2026-01-07T15:23:46Z     INFO    Starting reconciliation {"controller": "talosupgrade", "controllerGroup": "tuppr.home-operations.com", "controllerKind": "TalosUpgrade", "TalosUpgrade": {"name":"talos"}, "namespace": "", "name": "talos", "reconcileID": "5ba2deca-3989-40a3-9882-8da270105d00", "talosupgrade": "talos"}
2026-01-07T15:23:46.164606370Z 2026-01-07T15:23:46Z     DEBUG   Starting upgrade processing     {"controller": "talosupgrade", "controllerGroup": "tuppr.home-operations.com", "controllerKind": "TalosUpgrade", "TalosUpgrade": {"name":"talos"}, "namespace": "", "name": "talos", "reconcileID": "5ba2deca-3989-40a3-9882-8da270105d00", "talosupgrade": "talos", "generation": 3}
2026-01-07T15:23:46.164653818Z 2026-01-07T15:23:46Z     DEBUG   Upgrade already completed or failed     {"controller": "talosupgrade", "controllerGroup": "tuppr.home-operations.com", "controllerKind": "TalosUpgrade", "TalosUpgrade": {"name":"talos"}, "namespace": "", "name": "talos", "reconcileID": "5ba2deca-3989-40a3-9882-8da270105d00", "talosupgrade": "talos", "generation": 3, "phase": "Completed"}
2026-01-07T15:25:58.318964941Z 2026-01-07T15:25:58Z     INFO    Starting KubernetesUpgrade reconciliation       {"controller": "kubernetesupgrade", "controllerGroup": "tuppr.home-operations.com", "controllerKind": "KubernetesUpgrade", "KubernetesUpgrade": {"name":"kubernetes"}, "namespace": "", "name": "kubernetes", "reconcileID": "d4cdae99-3a78-4e57-b930-f6ba4deae872", "kubernetesupgrade": "kubernetes"}
2026-01-07T15:25:58.319395443Z 2026-01-07T15:25:58Z     DEBUG   Starting Kubernetes upgrade processing  {"controller": "kubernetesupgrade", "controllerGroup": "tuppr.home-operations.com", "controllerKind": "KubernetesUpgrade", "KubernetesUpgrade": {"name":"kubernetes"}, "namespace": "", "name": "kubernetes", "reconcileID": "d4cdae99-3a78-4e57-b930-f6ba4deae872", "kubernetesupgrade": "kubernetes", "generation": 4}
2026-01-07T15:25:58.319444687Z 2026-01-07T15:25:58Z     DEBUG   Kubernetes upgrade already completed or failed  {"controller": "kubernetesupgrade", "controllerGroup": "tuppr.home-operations.com", "controllerKind": "KubernetesUpgrade", "KubernetesUpgrade": {"name":"kubernetes"}, "namespace": "", "name": "kubernetes", "reconcileID": "d4cdae99-3a78-4e57-b930-f6ba4deae872", "kubernetesupgrade": "kubernetes", "generation": 4, "phase": "Completed"}

Originally posted by @tscibilia in #65

[onedr0p]

The current logic that should prevent this from happening doesn't appear to working great.

[tscibilia]

I don't believe I am. I just copied the last few lines of the logs.
I asked Claude to provide some additional info that might be helpful? Take it for what it's worth (I don't actually know how tuppr's logic works), please LMK if you need anything else.

Timeline

1. PR merged at 2026-01-07T13:52:35Z updating TalosUpgrade spec from v1.12.0 → v1.12.1
2. TalosUpgrade resource updated at 2026-01-07T13:53:46Z (generation incremented to 3)
3. Both upgrade resources remained in Completed phase for ~30 minutes until manual intervention

State When Issue Occurred

TalosUpgrade Resource

metadata:
  generation: 3  # Updated
status:
  observedGeneration: 3  # Seen by controller
  phase: Completed
  message: Successfully upgraded 0 nodes  # ⚠️ Completed, but with 0 nodes
  completedNodes: []
  currentNode: ""
  failedNodes: []
  lastUpdated: "2026-01-07T13:53:46Z"

KubernetesUpgrade Resource

metadata:
  generation: 4
status:
  observedGeneration: 4
  phase: Completed 

Controller Behavior

The logs show the controller entered an early-exit loop:

DEBUG  Starting upgrade processing  {"generation": 3}
DEBUG  Upgrade already completed or failed  {"generation": 3, "phase": "Completed"}

Summary

1. Generation increments -> controller sees the new generation
2. Resource is already in Completed phase, controller appears to:
   • Update observedGeneration to match new generation
   • Keeps phase: Completed
   • Reports "Successfully upgraded 0 nodes"

This suggests the "already completed" check happens before determining if the new spec requires work.

[onedr0p]

The log lines in the issue description looks like the talos upgrade failed and then the kubernetes upgrade failed.

[tscibilia]

Oh I see what you mean, there was no Kubernetes upgrade at the time but the logs just keep repeating like that.

If that's not normal, what should I be seeing?

[onedr0p]

Maybe it is normal 😄 and if that's the case I am unsure what is happening, anyways I am going to leave this issue open as there is a problem with upgrades running at conflicting times. Feel free to open a new issue with information on your problem too.