Archive Workflow node retry unexpectedly reruns successful TaskGroup children when template parameters are overridden

[KollenHub]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Archive Workflow retry cannot refresh templates without re-expanding successful TaskGroups

Summary

The Archive Workflow retry API cannot retry a single node with a newer container image or resource configuration while preserving successful TaskGroup children.

The current workaround is to pass image parameters to Archive Retry. In Argo Workflows 4.1.2, any non-empty retry parameter list causes successful TaskGroup/StepGroup children on the reset path to be deleted and re-expanded. As a result, retrying a failed downstream node can unexpectedly recreate and rerun previously successful fan-out tasks.

Environment

• Argo Workflows: v4.1.2
• API: PUT /api/v1/archived-workflows/{archiveUID}/retry
• Workflow type: DAG containing a TaskGroup created by withParam

Reproduction

Assume an archived workflow has this execution state:

shard-api          Succeeded
frame-processing   TaskGroup Succeeded
  frame-processing(0)  Succeeded
post-processing    Failed

The archived workflow has the original image in its templates. A new image is required for the retry.

Send a node-scoped archive retry request:

PUT /api/v1/archived-workflows/<archive-uid>/retry
Content-Type: application/json

{
  "restartSuccessful": false,
  "nodeFieldSelector": "id=<post-processing-node-id>",
  "parameters": [
    "cpu_image=registry.example/worker:new"
  ]
}

The parameter is needed today because there is no archive API for overriding the archived Workflow's template image/resources before retry.

Actual behavior

The selected post-processing node is reset, but the successful frame-processing TaskGroup child is deleted and recreated. The recreated frame Pod runs again, even though it was not selected and restartSuccessful is false.

This is caused by the parameter-handling branch in FormulateRetryWorkflow. In Argo Workflows 4.1.2, when parameters is non-empty, the retry implementation deletes children of reset TaskGroup/StepGroup nodes so that they can be expanded with new parameters.

Relevant source code:

workflow/util/util.go, FormulateRetryWorkflow:

// Delete children of TaskGroup/StepGroup nodes being reset when parameters are overridden
if len(parameters) > 0 {
    for nodeID := range toReset {
        ...
        if n.Type == wfv1.NodeTypeTaskGroup || n.Type == wfv1.NodeTypeStepGroup {
            ...
            for childID := range getChildren(dagNode) {
                toDelete[childID] = true
            }
        }
    }
}

Expected behavior

There should be a way to apply new container images and resource requests/limits to an archived Workflow retry without treating those changes as workflow input parameter changes.

For the reproduction above, the expected result is:

shard-api          remains Succeeded
frame-processing   remains Succeeded; its existing child Pod is not recreated
post-processing    is recreated with the new image/resources

The same behavior should work when the selected node is successful and restartSuccessful is explicitly true.

Suggested API design

Add a separate template/resource override field to the retry request, distinct from business parameter overrides:

{
  "restartSuccessful": false,
  "nodeFieldSelector": "id=<post-processing-node-id>",
  "templateOverrides": {
    "run-post-processing": {
      "image": "registry.example/worker:new",
      "resources": {
        "requests": {"cpu": "10", "memory": "16Gi"},
        "limits": {"cpu": "16", "memory": "32Gi"}
      }
    }
  }
}

The implementation could apply templateOverrides to the archived Workflow copy before calling the existing retry formulation logic. templateOverrides should not enable the len(parameters) > 0 TaskGroup child deletion path. Existing parameters behavior should remain unchanged for workflows that intentionally change withParam or other business inputs.

An alternative would be a separate specPatch/template override API for archived workflows, with validation that only template images and resource fields can be changed.

Why this matters

Without this capability, operators have to choose between:

1. Retrying an archived node with the old image/resources; or
2. Creating a new Workflow from scratch, which loses the original successful-node state and cannot provide node-scoped retry semantics.

A template/resource override would make archive retry equivalent to live Workflow retry while preserving the original execution graph and successful node state.

Workaround

For a live Workflow, updating spec.templates[*].container.image/resources first and then calling Retry without parameters avoids the problem. This workaround is not available after the Workflow has been archived because the Archive API does not expose a supported mutation operation for the archived Workflow spec.

Version(s)

v4.1.2

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

The following is a minimal workflow shape. `fanout` creates a successful
`withParam` TaskGroup and `target` fails after it:


apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: archive-retry-repro-
spec:
  entrypoint: main
  arguments:
    parameters:
      - name: worker_image
        value: busybox:1.36
  templates:
    - name: main
      dag:
        tasks:
          - name: produce
            template: produce
          - name: fanout
            template: fanout
            depends: produce.Succeeded
            withParam: "{{tasks.produce.outputs.parameters.items}}"
          - name: target
            template: target
            depends: fanout.Succeeded
    - name: produce
      script:
        image: busybox:1.36
        command: [sh]
        source: 'printf "[\\"0\\"]" > /tmp/items.json'
      outputs:
        parameters:
          - name: items
            valueFrom:
              path: /tmp/items.json
    - name: fanout
      container:
        image: "{{workflow.parameters.worker_image}}"
        command: [sh, -c]
        args: ["echo fanout-success"]
    - name: target
      container:
        image: "{{workflow.parameters.worker_image}}"
        command: [sh, -c]
        args: ["echo target-fails; exit 1"]


Submit this Workflow, wait until it is `Failed`, and ensure the Workflow has
been archived. Record the archive UID, the new archive-retry Workflow name,
and the node ID of `target` from `status.nodes`.

The archived Workflow should have this execution state:


shard-api          Succeeded
frame-processing   TaskGroup Succeeded
  frame-processing(0)  Succeeded
post-processing    Failed


The archived workflow has the original image in its templates. A new image is required for the retry.

Send a node-scoped archive retry request:


PUT /api/v1/archived-workflows/<archive-uid>/retry
Content-Type: application/json



{
  "restartSuccessful": false,
  "nodeFieldSelector": "id=<post-processing-node-id>",
  "parameters": [
    "cpu_image=registry.example/worker:new"
  ]
}


The parameter is needed today because there is no archive API for overriding the archived Workflow's template image/resources before retry.

For example:


curl -k -X PUT "$ARGO_SERVER/api/v1/archived-workflows/$ARCHIVE_UID/retry" \
  -H 'Content-Type: application/json' \
  -d "$(jq -n --arg node "$TARGET_NODE_ID" '{
    restartSuccessful: false,
    nodeFieldSelector: ("id=" + $node),
    parameters: ["worker_image=busybox:1.37"]
  }')"


Inspect the new Workflow returned by the request:


kubectl -n argo get workflow "$NEW_WORKFLOW_NAME" -o json \
  | jq '.status.nodes | to_entries[] | {name: .value.displayName, type: .value.type, phase: .value.phase, startedAt: .value.startedAt}'


The `fanout` child has a new execution timestamp even though only `target` was
selected for retry.

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[emagana]

I will give it a try to this one