fix(controller): prevent crash-looping pods from resetting progress deadline. Fixes #3765#4769
Conversation
…eadline Split RolloutProgressing() into RolloutScalingProgress() (structural changes like replica scaling, step advancement, selector switches) and RolloutReadinessProgress() (ReadyReplicas/AvailableReplicas increases). When the Progressing condition already exists with ReplicaSetUpdatedReason, readiness-only progress no longer resets LastUpdateTime. This prevents crash-looping pods from indefinitely deferring ProgressDeadlineExceeded through ReadyReplicas/AvailableReplicas oscillation. Signed-off-by: Yurii Popivniak <ypopivniak@cargurus.com>
Signed-off-by: Yurii Popivniak <ypopivniak@cargurus.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4769 +/- ##
==========================================
+ Coverage 84.99% 85.01% +0.02%
==========================================
Files 164 164
Lines 18998 19005 +7
==========================================
+ Hits 16147 16158 +11
+ Misses 1998 1996 -2
+ Partials 853 851 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Published E2E Test Results 4 files 4 suites 3h 38m 54s ⏱️ For more details on these failures, see this check. Results for commit a262722. ♻️ This comment has been updated with latest results. |
Published Unit Test Results2 479 tests 2 479 ✅ 3m 21s ⏱️ Results for commit a262722. ♻️ This comment has been updated with latest results. |
|



When pods crash-loop,
ReadyReplicas/AvailableReplicasoscillate (e.g. 91→92→91→92) as pods briefly start before crashing again. Each upswing is treated as progress byRolloutProgressing(), resetting theProgressingcondition'sLastUpdateTimeand preventingProgressDeadlineExceededfrom ever firing.This PR splits
RolloutProgressing()into two functions:RolloutScalingProgress()— structural changes (replica scaling, step advancement, selector switches). Always resets the deadline.RolloutReadinessProgress()—ReadyReplicas/AvailableReplicasincreases. Does NOT reset the deadline when theProgressingcondition already exists withReplicaSetUpdatedReason.Applies to both Canary and BlueGreen strategies.
Fixes: #3765
Current behavior:
Rollout gets stuck in
Progressingphase indefinitely when canary pods crash-loop, because replica readiness oscillation keeps resetting the progress deadline clock.Expected behavior:
After
progressDeadlineSecondselapses without structural progress, the Rollout transitions toDegradedphase withProgressDeadlineExceededreason.Checklist:
"fix(controller): Updates such and such. Fixes #1234".