Skip to content

fix(controller): sync status when traffic routing errors to allow progress deadline checks#4826

Open
Goutham-Annem wants to merge 3 commits into
argoproj:masterfrom
Goutham-Annem:fix/4626-progress-deadline-traffic-routing
Open

fix(controller): sync status when traffic routing errors to allow progress deadline checks#4826
Goutham-Annem wants to merge 3 commits into
argoproj:masterfrom
Goutham-Annem:fix/4626-progress-deadline-traffic-routing

Conversation

@Goutham-Annem

Copy link
Copy Markdown

What this PR does

When reconcileTrafficRouting() returns an error — for example, when Istio delays the DestinationRule switch until the canary ReplicaSet is fully available — the controller previously returned immediately without calling syncRolloutStatusCanary(). Because syncRolloutStatusCanary() is where evaluateProgressDeadlineAbort() runs, the progress deadline could never fire while traffic routing kept failing on every reconciliation cycle.

Root cause (from issue #4626):

rolloutCanary()
  └─► reconcileTrafficRouting()  ← returns error every cycle
  └─► return err                 ← EXITS HERE
  ✗  syncRolloutStatusCanary()   ← NEVER REACHED
       └─► evaluateProgressDeadlineAbort()  ← NEVER FIRES

Fix: When reconcileTrafficRouting() fails, call syncRolloutStatusCanary() before returning the error. Status is updated on every reconciliation cycle so the deadline check always gets a chance to run.

if trafficErr := c.reconcileTrafficRouting(); trafficErr != nil {
    if syncErr := c.syncRolloutStatusCanary(); syncErr != nil {
        return syncErr
    }
    return trafficErr
}

Test coverage

  • Added TestProgressDeadlineAbortWithTrafficRoutingError: creates a canary rollout with progressDeadlineAbort: true, injects an already-timed-out progressing condition, sets the mock traffic router to return an error, and verifies that verifyPatchedRolloutAborted() fires even when the traffic routing error path is taken.
  • Updated two existing tests (TestReconcileTrafficRoutingSetWeightErr, TestReconcileTrafficRoutingVerifyWeightEndOfRollout, TestTrafficRoutingErrorsWhenNewCanaryHasNoReplicas) to expect the extra status patch that is now emitted on every traffic routing error.

Which issue(s) this PR fixes

Fixes #4626

Note: This PR was developed with AI assistance (Claude Code).

…ss deadline checks

When `reconcileTrafficRouting()` returns an error (e.g. Istio
delaying the DestinationRule switch until the canary ReplicaSet is
fully available), the original code returned immediately, so
`syncRolloutStatusCanary()` — and its `evaluateProgressDeadlineAbort()`
call — was never reached on any reconciliation cycle.

This meant a rollout with `progressDeadlineAbort: true` and canary
pods stuck in CrashLoopBackOff would never be aborted, regardless
of how much time had elapsed past `progressDeadlineSeconds`.

Fix: when `reconcileTrafficRouting()` fails, call
`syncRolloutStatusCanary()` before returning the error. Status is
updated on every cycle, so the deadline check always fires.

Fixes argoproj#4626

Signed-off-by: Goutham Annem <gouthemannem@gmail.com>
@Goutham-Annem Goutham-Annem requested a review from a team as a code owner July 5, 2026 23:11
@codecov

codecov Bot commented Jul 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.04%. Comparing base (d1c2967) to head (29b4ddb).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4826      +/-   ##
==========================================
- Coverage   85.09%   85.04%   -0.05%     
==========================================
  Files         166      166              
  Lines       19136    19139       +3     
==========================================
- Hits        16283    16277       -6     
- Misses       2005     2010       +5     
- Partials      848      852       +4     
Flag Coverage Δ
e2e 52.78% <60.00%> (-0.20%) ⬇️
unit-tests 81.40% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Published E2E Test Results

  4 files    4 suites   3h 53m 25s ⏱️
122 tests 111 ✅  7 💤  4 ❌
498 runs  460 ✅ 28 💤 10 ❌

For more details on these failures, see this check.

Results for commit 29b4ddb.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Published Unit Test Results

2 518 tests   2 518 ✅  3m 23s ⏱️
  130 suites      0 💤
    1 files        0 ❌

Results for commit 29b4ddb.

♻️ This comment has been updated with latest results.

@Goutham-Annem Goutham-Annem changed the title fix(rollout): sync status when traffic routing errors to allow progress deadline checks fix(controller): sync status when traffic routing errors to allow progress deadline checks Jul 10, 2026
…nd status sync fail

Adds TestProgressDeadlineSyncStatusErrorWithTrafficRoutingError to exercise the branch
where reconcileTrafficRouting() and syncRolloutStatusCanary() both return errors, verifying
that the sync error (not the traffic routing error) is surfaced so the controller requeues.

Signed-off-by: Goutham Annem <gouthemannem@gmail.com>
@Goutham-Annem Goutham-Annem force-pushed the fix/4626-progress-deadline-traffic-routing branch from 8cad80f to ee830c2 Compare July 10, 2026 11:27
TestProgressDeadlineAbortWithTrafficRoutingError and
TestProgressDeadlineSyncStatusErrorWithTrafficRoutingError share identical fixture
setup. Extract into newProgressDeadlineTrafficRoutingFixture to stay below
SonarCloud's 40% new-code duplication threshold.

Signed-off-by: Goutham Annem <gouthemannem@gmail.com>
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant