Rolling Update Deadlock in PodClique Controller

[gflarity]

Summary

Rolling updates can get permanently stuck due to a race condition between when UpdateEndedAt is set and when pods are counted. The test Test_RU9_RollingUpdateAllPodCliques fails intermittently with a 4-minute timeout waiting for the rolling update to complete.

What Should Happen

1. Update in progress: Old pods are deleted, new pods are created.

   • CurrentPodTemplateHash = OLD hash
   • RollingUpdateProgress.PodTemplateHash = NEW hash
   • UpdateEndedAt = nil → mutateUpdatedReplica() counts against NEW hash ✓
   • UpdatedReplicas increases as new pods become Ready

2. All new pods become Ready:

   • UpdatedReplicas = 2, Replicas = 2
   • All old pods have been deleted

3. markRollingUpdateEnd() sets UpdateEndedAt:

   • UpdateEndedAt is now set → IsPCLQUpdateInProgress() returns false
   • At this moment: UpdatedReplicas = 2, Replicas = 2 ✓

4. Next status reconciliation — hash switch happens safely:

   mutateCurrentHashes() runs first:

   • IsPCLQUpdateInProgress()? → NO (because UpdateEndedAt is set)
   • UpdatedReplicas == Replicas? → YES (2 == 2) ✓
   • Updates CurrentPodTemplateHash to the NEW hash

   mutateUpdatedReplica() runs next:

   • IsPCLQUpdateInProgress()? → NO (because UpdateEndedAt is set)
   • Uses CurrentPodTemplateHash (now the NEW hash!)
   • Counts pods matching NEW hash → 2 pods match ✓
   • UpdatedReplicas = 2

5. Update completes successfully.

What Happened (The Bug)

1. Update in progress: Old pods are deleted, new pods are created.

   • CurrentPodTemplateHash = OLD hash
   • RollingUpdateProgress.PodTemplateHash = NEW hash
   • UpdateEndedAt = nil → mutateUpdatedReplica() counts against NEW hash ✓
   • UpdatedReplicas = 1 (one new pod is Ready)

2. Last old pod deleted, but one new pod is still becoming Ready:

   • markRollingUpdateEnd() is called (no old pods remain)
   • Sets UpdateEndedAt → IsPCLQUpdateInProgress() now returns false
   • At this moment: UpdatedReplicas = 1, Replicas = 2

3. Next status reconciliation — the breaking moment:

   mutateCurrentHashes() runs first:

   • IsPCLQUpdateInProgress()? → NO (because UpdateEndedAt is set)
   • UpdatedReplicas == Replicas? → NO (1 != 2)
   • Refuses to update CurrentPodTemplateHash — it stays as the OLD hash

   mutateUpdatedReplica() runs next:

   • IsPCLQUpdateInProgress()? → NO (because UpdateEndedAt is set)
   • Uses CurrentPodTemplateHash (the OLD hash!)
   • Counts pods matching OLD hash → 0 pods match (all pods have NEW hash)
   • Sets UpdatedReplicas = 0

4. Deadlock — every subsequent reconciliation:

   • mutateCurrentHashes(): UpdatedReplicas (0) != Replicas (2) → refuses to update hash
   • mutateUpdatedReplica(): uses OLD hash → counts 0 pods → UpdatedReplicas = 0
   • UpdatedReplicas stays at 0 forever

The system is permanently stuck. CurrentPodTemplateHash will never be updated because UpdatedReplicas != Replicas, but UpdatedReplicas will never be correct because it's counting against the wrong hash.

Evidence

e2eRU9-v2.diag.txt

nce from Diagnostics

1. PodClique Shows Contradictory State

# From diagnostic dump
status:
  replicas: 2
  updatedReplicas: 0                                    # ← Should be 2!
  currentPodCliqueSetGenerationHash: f6c5fd949cb444c6558  # ← OLD hash
  currentPodTemplateHash: 9f945b7b659c5f4cb8f             # ← OLD hash (never updated)
  rollingUpdateProgress:
    podCliqueSetGenerationHash: c947687c4d79f9cfb59       # ← NEW target hash
    podTemplateHash: 574f6fb86d49bfdfbd9d                 # ← NEW target hash
    updateStartedAt: "2026-01-13T00:38:19Z"
    updateEndedAt: "2026-01-13T00:38:19Z"                 # ← Update marked COMPLETE!

The contradiction:

• updateEndedAt is set → Rolling update component thinks it's done
• currentPodTemplateHash has OLD value → Hash was never updated
• updatedReplicas: 0 → Pods being counted against wrong hash

2. Controller Logs Show the Loop

{
  "level": "info",
  "ts": "2026-01-13T00:38:29.260Z",
  "logger": "podclique-controller",
  "msg": "PodClique is currently updating, cannot set PodCliqueSet CurrentGenerationHash yet",
  "PodClique": {"name": "workload1-0-pc-a", "namespace": "default"}
}

This was logged 10 seconds after updateEndedAt was set. The controller is stuck refusing to update the hash because UpdatedReplicas != Replicas.

3. Timeline of Events

Time	Event	State
00:38:18	Rolling update triggered	updateStartedAt set
00:38:19	updateEndedAt set	Old pods gone, but UpdatedReplicas may not equal Replicas
00:38:19+	First broken reconciliation	mutateUpdatedReplica() switches to OLD hash, counts 0
00:38:29	Controller logs "cannot set hash"	Deadlock confirmed
00:42:18	Test timeout	4 minutes with no progress

[gflarity]

I spent time analyzing the diagnostic logs from another reproduction of this issue, and I now understand what's happening. However, I'm not sure if this is a bug or a design issue.

The test expects that when additional nodes become available, the two scaled PCSG replicas (scaled gangs) should each achieve their MinAvailable and get scheduled. Instead, the base gang's "elastic" pods — pods beyond MinAvailable — get scheduled first, consuming the available nodes before the scaled gangs become visible to Kai.

This happens because of a race condition. When the base gang achieves MinAvailable, Grove begins removing scheduling gates from the scaled gang pods. Meanwhile, the test uncordons more nodes. If Kai runs a scheduling cycle while the gates are still being removed, it sees only the base gang (scaled gangs are still gated and therefore invisible) and allocates its elastic pods. By the time the gates are removed, the resources are exhausted.

The deeper question is whether this is actually incorrect behaviour. The base gang contains "elastic" pods because its total replicas exceed MinAvailable. The scaled gangs represent additional PCSG replicas, which are themselves a form of elastic scaling. So should the base gang's elastic pods really take priority over the scaled gangs' MinAvailable? It's not obvious which should win when resources are limited. But currently, neither intentionally wins — the outcome is determined entirely by timing, which seems wrong regardless of what the "correct" priority should be.

See the walkthrough below for details.
@danbar2 @sanjaychatterjee @nvrohanv @athreesh

GS8_diag.txt

Test_GS8 Workload Structure

From operator/e2e/yaml/workload2.yaml:

Component	Replicas	MinAvailable
pc-a	2	1
PCSG sg-x	2→3	1
└─ pc-b	1	1
└─ pc-c	3	1

PodGang Structure (simplified):

PodGang: workload2-0  (Base Gang)
├── PodGroup: workload2-0-pc-a         → 2 pods, minReplicas: 1
├── PodGroup: workload2-0-sg-x-0-pc-b  → 1 pod,  minReplicas: 1
└── PodGroup: workload2-0-sg-x-0-pc-c  → 3 pods, minReplicas: 1
    Total: 6 pods, MinAvailable sum: 3

PodGang: workload2-0-sg-x-1  (Scaled Gang 1)
    labels: grove.io/base-podgang: workload2-0  ← depends on base
├── PodGroup: workload2-0-sg-x-1-pc-b  → 1 pod,  minReplicas: 1
└── PodGroup: workload2-0-sg-x-1-pc-c  → 3 pods, minReplicas: 1
    Total: 4 pods, MinAvailable sum: 2

PodGang: workload2-0-sg-x-2  (Scaled Gang 2)
    labels: grove.io/base-podgang: workload2-0  ← depends on base
├── PodGroup: workload2-0-sg-x-2-pc-b  → 1 pod,  minReplicas: 1
└── PodGroup: workload2-0-sg-x-2-pc-c  → 3 pods, minReplicas: 1
    Total: 4 pods, MinAvailable sum: 2

Race Condition Timeline

1. Limited resources initially

• Nodes are cordoned or resources are constrained
• Only enough capacity for the base gang's MinReplicas

2. Base gang achieves MinReplicas

• Base gang gets scheduled (e.g., 3 pods on 3 nodes)
• Grove detects this and begins gate removal for scaled gangs

3. Gate removal begins

• Grove starts patching pods to remove scheduling gates
• This is not instantaneous — each pod requires an API call

4. Test uncordons more nodes

• Test sees base gang is ready and adds more resources
• Test expects scaled gangs to now schedule
• Test doesn't wait for gate removal to complete

5. Gate removal completes

• All scaled gang pods are now ungated and eligible for scheduling

What The Test Expects

When resources become available, Kai's fair scheduling will ensure all gangs achieve their MinAvailable before any gang gets "elastic" (beyond-MinAvailable) pods. With 7 nodes available:

• Base gang: 3 pods (MinAvailable) ✓
• Scaled gang 1: 2 pods (MinAvailable) ✓
• Scaled gang 2: 2 pods (MinAvailable) ✓
• Total: 7 pods scheduled, each gang satisfied

The remaining 7 pods (base gang's 3 elastic + scaled gangs' 4 elastic) would be scheduled later as more resources become available.

What Happens Sometimes

Because scaled gangs are still gated when Kai runs, they're invisible. Kai sees only the base gang and allocates its elastic pods:

• Base gang: 6 pods (3 MinAvailable + 3 elastic) — consumes 6 of 7 nodes
• Scaled gang 1: 0 pods — filtered out, then can't fit MinAvailable (needs 2, only 1 node left)
• Scaled gang 2: 0 pods — filtered out, then can't fit MinAvailable (needs 2, only 1 node left)

The base gang "starves" the scaled gangs because they weren't visible during allocation.

Kai scheduler sees:

• New nodes available
• Base gang: 3 scheduled + 3 pending (ungated) pods — passes IsReadyForScheduling(), included in scheduling queue
• Scaled gang 1: 4 gated pods — fails IsReadyForScheduling() (0 ready < 2 MinAvailable), filtered out
• Scaled gang 2: 4 gated pods — fails IsReadyForScheduling() (0 ready < 2 MinAvailable), filtered out

Kai scheduler does:

• Filters out scaled gangs (not ready)
• Allocates to base gang (already has MinReplicas → 1 pod at a time)