Add retryStrategy.maxExecutionDuration to bound cumulative Pod execution time

[herikwebb]

Summary

retryStrategy.backoff.maxDuration is a wall-clock deadline measured from the first attempt. It includes Pod scheduling, image pulling, initialization, retry back-off, and execution time. On congested clusters, infrastructure latency can therefore exhaust the deadline before the configured retries run.

Add an optional retryStrategy.maxExecutionDuration field that limits cumulative application execution across retry attempts without charging infrastructure and back-off delays against the budget.

retryStrategy:
  limit: "5"
  maxExecutionDuration: 30m

Proposed semantics:

• Count completed, failed or errored attempts for Pod-backed templates: container, script, containerSet, resource, and data.
• Measure each attempt as one wall-clock execution envelope from the earliest observed main-container start to the latest main-container finish.
  • For containerSet, overlapping containers are not double-counted, while gaps within that envelope do count.
  • If the final container timestamp is unavailable, conservatively count from the observed main-container start until Argo observes the attempt complete.
• Exclude Pod scheduling/Pending time, image pulling, init containers, post-main output processing, and retry back-off waits.
• Attempts that fail before any main container starts consume no execution budget.
• Evaluate the cumulative budget only after an attempt fails or errors and before another retry is created.
• Do not terminate an active attempt. Accept a successful attempt even if it causes the cumulative duration to reach or exceed the configured budget.
• Resolve parameterized values when the retry sequence starts and retain that resolved value across attempts and controller reconciliations. Attempt-scoped retries and lastRetry variables are therefore not supported in this field.
• Reject a directly configured value on a non-Pod template. When inherited through workflow-level retry settings or templateDefaults, ignore this field for non-Pod templates while preserving the other retry settings.
• Keep this budget independent of backoff.maxDuration, pendingTimeout, and activeDeadlineSeconds. If more than one limit is configured, whichever prevents the next retry first takes effect.

This is an additive, opt-in field.
Existing Workflows retain their current behavior, and the semantics of backoff.maxDuration do not change.
Because attempts that never start consume no execution budget, users should combine this setting with limit, pendingTimeout, or backoff.maxDuration when they also need a wall-clock bound.

I would like to contribute this change after agreement on the API and accounting semantics.

Related work:

• Retrying workflows have short deadline ignoring activeDeadlineSeconds #13044 and its documentation fix docs: Clarify maxDuration behavior when used with activeDeadlineSeconds. Fixes #13044 #13068 describe the existing wall-clock behavior of backoff.maxDuration.
• Add a pendingTimeout parameter #10341 and its implementation feat: add pendingTimeout for non-deadline timeout Fixes #10341 #12762 added pendingTimeout, which is complementary when users also need to bound Pod startup time.
• Add a metric for retries suppressed by duration budgets #16856 proposes aggregate telemetry for retries suppressed by duration budgets. If both proposals are accepted, execution-budget exhaustion would use a bounded MaxExecutionDurationExceeded reason on that metric.
• docs(retries): clarify maxDuration retry deadline #16855 clarifies the existing backoff.maxDuration semantics in the retries guide.

Use Cases

• Allow retries to remain effective during node scale-up, CNI throttling, image pulls, or scheduler congestion.
• Limit how much failed application execution a retry sequence may consume without terminating a currently running attempt.
• Apply a stable execution budget across several retries while retaining a separate end-to-end wall-clock deadline when needed.

Alternatives Considered

• Change backoff.maxDuration to exclude Pending and back-off time. This would be a breaking semantic change for existing Workflows.
• Add a different anchor such as maxDurationFrom: ContainerStart. A single anchor would still count later Pending and back-off intervals. Pausing and resuming the existing deadline would also overload a field that currently represents a wall-clock budget.
• Use activeDeadlineSeconds. That can terminate an active attempt and does not represent cumulative failed execution across retries.

Acceptance Criteria

• The API, generated schemas, field documentation, and an example expose retryStrategy.maxExecutionDuration.
• Values must resolve to a positive duration before the first attempt starts.
• The resolved budget and observed execution timing survive controller reconciliations.
• Cumulative accounting follows the inclusion and exclusion rules above, including ContainerSet execution envelopes and attempts whose main container never starts.
• Reaching the budget suppresses only a subsequent retry; it never terminates an active or successful attempt.
• Existing retry behavior is unchanged when the field is omitted.

---

Message from the maintainers:

Love this feature request? Give it a 👍. We prioritise the proposals with the most 👍.