fix(trafficrouting): do not block abort/progressDeadline when Istio delays DestinationRule switch. Fixes #4626

[ChanghwanK]

Fixes #4626

Summary

When subset-level Istio traffic routing delays the DestinationRule switch because a traffic-receiving ReplicaSet is not fully available (behavior introduced by #4612), the resulting error propagated as fatal and short-circuited rolloutCanary() before it ever reached syncRolloutStatusCanary(). As a result, a rollout whose canary pods never become available could not time out, turn Degraded, or auto-abort: the controller retried the same reconcile forever (rate-limited requeue) and the rollout stayed Progressing indefinitely, requiring manual intervention (and in the worst observed case, a controller restart) to recover.

The core of the bug is a circular blocking condition: the condition that should trigger the progress-deadline abort (an unavailable ReplicaSet) is exactly the condition that blocked the code path that evaluates the abort.

Root cause

Call path (all line refs at current master):

1. rollout/trafficrouting/istio/istio.go UpdateHash() intentionally skips the DestinationRule switch and returns a plain error when shouldDelayDestinationRuleUpdate() is true. This is an expected, transient condition (logged at Info level), but it is indistinguishable from a fatal error to callers.
2. rollout/trafficrouting.go reconcileTrafficRouting() propagates it unchanged.
3. rollout/canary.go:67-69 rolloutCanary() returns early, skipping everything after it: experiments, analysis runs, ReplicaSet scaling/cleanup, and critically syncRolloutStatusCanary() → persistRolloutStatus(), which is where evaluateProgressDeadlineAbort(), Progressing-condition timeout evaluation, and requeueStuckRollout() live.

Since the delay error recurs on every reconcile as long as the ReplicaSet stays unavailable (e.g. crashlooping pods), the abort/timeout evaluation is blocked permanently, not transiently. Note that abort is also the natural exit for this loop: shouldDelayDestinationRuleUpdate() already exempts the canary ReplicaSet during an abort, so an abort would make the delay condition disappear and let the system converge — it just could never fire.

Fix

Three small pieces:

1. rollout/trafficrouting/istio/istio.go: wrap the delay error with a sentinel (ErrDestinationRuleUpdateDelayed, %w) so callers can distinguish this expected condition via errors.Is. The error message is unchanged. All other errors (API failures etc.) are untouched and keep the existing propagate-and-requeue behavior.
2. rollout/trafficrouting.go reconcileTrafficRouting(): when the sentinel is detected after UpdateHash(), skip SetWeight() for this round (preserving the ordering guarantee from fix(trafficrouting): ensure DestinationRule is updated before SetWeight on rollback #4612: never set weight while the DestinationRule switch is pending) but return nil so the rest of the reconcile still runs — abort/progressDeadline evaluation, conditions/phase updates, ReplicaSet cleanup, and the deadline-based requeue.
3. rollout/canary.go completedCurrentCanaryStep() + a trafficRoutingDelayed flag on rolloutContext: while the switch is delayed, the current step is not treated as completed, so currentStepIndex cannot advance while the desired traffic weight has not actually been applied. This keeps the progression backpressure that the fatal error used to provide.

Behavior before/after while the DestinationRule switch is delayed:

	before	after
SetWeight() called	no	no (unchanged, #4612 guarantee kept)
step index advances	no (reconcile aborted)	no (explicit guard)
progressDeadline / abort evaluated	never	every reconcile
Progressing → TimedOut, phase Degraded	never	on deadline expiry
ReplicaSet cleanup, analysis reconcile	never	every reconcile
fatal traffic-routing errors (API failures etc.)	propagate + requeue	propagate + requeue (unchanged)

Testing

• New: TestCanaryProgressDeadlineAbortNotBlockedByDelayedDestinationRuleSwitch reproduces the reported symptom end-to-end at the controller level: a canary rollout with progressDeadlineAbort: true, stuck past its deadline with a partially-available canary ReplicaSet, must still auto-abort while UpdateHash reports the delay. It fails on master (no abort patch is produced) and passes with this fix.
• New: TestReconcileTrafficRoutingUpdateHashDelayedErr verifies the delayed error is absorbed, SetWeight is not called (unstubbed on the mock, so a regression panics), the step is not completed (currentStepIndex does not advance, no RolloutStepCompleted event).
• Updated: TestRollbackDestinationRuleBeforeSetWeight (the fix(trafficrouting): ensure DestinationRule is updated before SetWeight on rollback #4612 scenario test) still asserts SetWeight is not called on delay, and now also asserts reconciliation continues (stale canary RS cleanup proceeds in the same sync instead of the sync erroring out).
• Updated: TestUpdateHashNoReadyReplicaSets additionally pins the errors.Is contract on the sentinel.
• Each new/updated assertion was verified to fail with the fix reverted (and with the step-completion guard disabled), so they genuinely pin the behavior.
• go build ./..., full go test ./... (0 failures), and golangci-lint run ./rollout/... (0 issues) pass locally.

Production context

This was hit three times in production (2026-05-07, 2026-05-29, 2026-06-23) on v1.8.2; details posted in #4626. The third recurrence affected 8 rollouts simultaneously and required a controller restart to recover. In all cases the trigger was canary pods that could never become available, and the rollouts stayed Progressing with no abort, no Degraded phase, and no status updates until manual intervention.

---

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
• The title of the PR is (a) conventional with a list of types and scopes found here, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
• I've signed my commits with DCO
• My builds are green. Try syncing with master if they are not.
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• I have run all tests locally (including the flaky ones) and they pass on my workstation — full unit suite passes locally; the Istio E2E suite (TestIstioSuite) was not run locally and relies on CI.
• I have used LLM/AI/Agent tools for this PR but I am responsible for all code of this PR
• I understand what the code does and WHY/HOW it works in several scenarios
• I know if my code is just adding new functionality or changing old functionality for existing users — this changes error-handling behavior for the Istio subset-level delayed-switch path only; all other error paths are unchanged.
• My organization is added to USERS.md.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.94%. Comparing base (bc55226) to head (7c30ebb).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4823      +/-   ##
==========================================
- Coverage   85.04%   84.94%   -0.10%     
==========================================
  Files         166      166              
  Lines       19136    19142       +6     
==========================================
- Hits        16275    16261      -14     
- Misses       2009     2020      +11     
- Partials      852      861       +9     

Flag	Coverage Δ
e2e	52.66% <100.00%> (-0.18%)	⬇️
unit-tests	81.39% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Published E2E Test Results

  4 files    4 suites   3h 48m 16s ⏱️
122 tests 112 ✅  7 💤  3 ❌
500 runs  461 ✅ 28 💤 11 ❌

For more details on these failures, see this check.

Results for commit 7c30ebb.

♻️ This comment has been updated with latest results.

[github-actions]

Published Unit Test Results

2 518 tests   2 518 ✅  3m 24s ⏱️
  130 suites      0 💤
    1 files        0 ❌

Results for commit 7c30ebb.

♻️ This comment has been updated with latest results.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[ChanghwanK]

Hi @zachaller / @kostis-codefresh, would appreciate a look at this when you have bandwidth.

This fixes a bug we've hit three times in production (2026-05-07, 2026-05-29, 2026-06-23 on v1.8.2) where canary rollouts get permanently stuck Progressing with no abort, no Degraded phase, and no status update, requiring manual intervention (once even a controller restart) to recover. Details in #4626.

The fix is small and scoped (sentinel error to distinguish the transient Istio DestinationRule delay from fatal errors), all CI is green, and it includes a new test that reproduces the exact stuck-forever scenario end-to-end and fails on master without the fix.

Happy to address any review feedback quickly.