Skip to content

fix(controller): prevent crash-looping pods from resetting progress deadline. Fixes #3765#4769

Open
ypopivniak-cg wants to merge 3 commits into
argoproj:masterfrom
ypopivniak-cg:fix/progress-deadline-crashloop
Open

fix(controller): prevent crash-looping pods from resetting progress deadline. Fixes #3765#4769
ypopivniak-cg wants to merge 3 commits into
argoproj:masterfrom
ypopivniak-cg:fix/progress-deadline-crashloop

Conversation

@ypopivniak-cg

@ypopivniak-cg ypopivniak-cg commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

When pods crash-loop, ReadyReplicas/AvailableReplicas oscillate (e.g. 91→92→91→92) as pods briefly start before crashing again. Each upswing is treated as progress by RolloutProgressing(), resetting the Progressing condition's LastUpdateTime and preventing ProgressDeadlineExceeded from ever firing.

This PR splits RolloutProgressing() into two functions:

  • RolloutScalingProgress() — structural changes (replica scaling, step advancement, selector switches). Always resets the deadline.
  • RolloutReadinessProgress()ReadyReplicas/AvailableReplicas increases. Does NOT reset the deadline when the Progressing condition already exists with ReplicaSetUpdatedReason.

Applies to both Canary and BlueGreen strategies.

Fixes: #3765

Current behavior:
Rollout gets stuck in Progressing phase indefinitely when canary pods crash-loop, because replica readiness oscillation keeps resetting the progress deadline clock.

Expected behavior:
After progressDeadlineSeconds elapses without structural progress, the Rollout transitions to Degraded phase with ProgressDeadlineExceeded reason.

Checklist:

  • Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
  • The title of the PR is (a) conventional with a list of types and scopes found here, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
  • I've signed my commits with DCO
  • I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
  • My builds are green. Try syncing with master if they are not.
  • I have run all tests locally (including the flaky ones) and they pass on my workstation
  • I have used LLM/AI/Agent tools for this PR but I am responsible for all code of this PR
  • I understand what the code does and WHY/HOW it works in several scenarios
  • I know if my code is just adding new functionality or changing old functionality for existing users
  • My organization is added to USERS.md.

…eadline

Split RolloutProgressing() into RolloutScalingProgress() (structural changes
like replica scaling, step advancement, selector switches) and
RolloutReadinessProgress() (ReadyReplicas/AvailableReplicas increases).

When the Progressing condition already exists with ReplicaSetUpdatedReason,
readiness-only progress no longer resets LastUpdateTime. This prevents
crash-looping pods from indefinitely deferring ProgressDeadlineExceeded
through ReadyReplicas/AvailableReplicas oscillation.

Signed-off-by: Yurii Popivniak <ypopivniak@cargurus.com>
@ypopivniak-cg ypopivniak-cg requested a review from a team as a code owner June 8, 2026 12:47
Signed-off-by: Yurii Popivniak <ypopivniak@cargurus.com>
@codecov

codecov Bot commented Jun 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.01%. Comparing base (76a703a) to head (a262722).
⚠️ Report is 26 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4769      +/-   ##
==========================================
+ Coverage   84.99%   85.01%   +0.02%     
==========================================
  Files         164      164              
  Lines       18998    19005       +7     
==========================================
+ Hits        16147    16158      +11     
+ Misses       1998     1996       -2     
+ Partials      853      851       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Published E2E Test Results

  4 files    4 suites   3h 38m 54s ⏱️
120 tests 109 ✅  7 💤 4 ❌
484 runs  452 ✅ 28 💤 4 ❌

For more details on these failures, see this check.

Results for commit a262722.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Published Unit Test Results

2 479 tests   2 479 ✅  3m 21s ⏱️
  129 suites      0 💤
    1 files        0 ❌

Results for commit a262722.

♻️ This comment has been updated with latest results.

@sonarqubecloud

sonarqubecloud Bot commented Jun 8, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rollout can sometimes run indefinitely when new pods are in a crash loop

1 participant