Artifact GC: a failed GC attempt is recorded as processed and never retried, leaving the finalizer permanently

[Joibel]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag and can confirm the issue still exists on :latest (the code path is unchanged on main at 38a9522)
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself

What happened? What did you expect to happen?

When a workflow completes with an artifact GC strategy configured, the controller adds the
workflows.argoproj.io/artifact-gc finalizer and then, on the next reconcile, runs
processArtifactGCStrategy to create the WorkflowArtifactGCTasks and the GC pod.

That function marks the strategy as processed in a defer, so it runs whether or not the
function returned an error:

// workflow/controller/artifact_gc.go:127
func (woc *wfOperationCtx) processArtifactGCStrategy(ctx context.Context, strategy wfv1.ArtifactGCStrategy) error {
	defer func() {
		woc.wf.Status.ArtifactGCStatus.SetArtifactGCStrategyProcessed(strategy, true)
		woc.updated = true
	}()

If anything inside it fails (task create, pod create, template lookup), operate logs
failed to GC artifacts and returns, and the deferred persistUpdates writes
strategiesProcessed.<strategy>: true to the workflow. From then on
artifactGCStrategiesReady (artifact_gc.go:107) considers the strategy done and never attempts
it again. Nothing ever marks the artifacts deleted, so allArtifactsDeleted() stays false and the
finalizer is never removed.

forceFinalizerRemoval: true does not help. It removes the finalizer only when
AllArtifactGCPodsRecouped() is true, and that returns false when podsRecouped is nil
(pkg/apis/workflow/v1alpha1/workflow_types.go:1644). A pod is only added to podsRecouped
when it is recouped (artifact_gc.go:582), not when it is created, so a workflow whose GC pod was
never created, or was created but never finished, has an empty map and the force flag can never fire.

The workflow ends up:

• phase Succeeded/Failed, workflows.argoproj.io/completed=true, archived
• metadata.finalizers: [workflows.argoproj.io/artifact-gc]
• status.artifactGCStatus.strategiesProcessed.OnWorkflowCompletion: true, no podsRecouped
• if deleted, stuck in Terminating with OnWorkflowDeletion: true as well

It stays like that forever. Because reconciliationNeeded is true while the finalizer is present,
the controller reconciles it every resync period doing nothing, and it keeps its parallelism slot
in the throttler. In the case that led me here, roughly 500 such workflows in one namespace had
consumed the entire namespaceParallelism budget, and every new workflow in that namespace was
being marked Pending with "Workflow processing has been postponed because too many workflows are
already running" indefinitely.

Any transient failure is enough to trigger this. Observed causes so far:

• API server pressure during a controller restart storm (429s / timeouts on the task or pod create)
• an admission webhook or ValidatingAdmissionPolicy rejecting the GC pod (ArtifactGC failures due to denying pod webhooks are not properly reported #14355 describes this)
• a retry after an update conflict failing with metadata.ownerReferences.uid: Invalid value: "": must not be empty on the GC pod

Expected behaviour: a failed GC attempt should be retried on subsequent reconciles, ideally with
backoff. The strategy should only be recorded as processed once the tasks and pod actually exist.
If the attempts are exhausted, the workflow should carry an ArtifactGCError condition so
forceFinalizerRemoval (or an operator) can release it, rather than silently holding the finalizer.

Related symptoms already filed: #10840, #12316, #14355, #16599. Each describes one trigger for the
same end state.

Version(s)

v4.0.1 observed in the field. Code path is identical on main.

Paste a minimal workflow that reproduces the issue

Any workflow with an output artifact and artifactGC.strategy: OnWorkflowCompletion, run on a
cluster where GC pod creation fails once. The Kyverno policy in #14355 is a reliable way to make the
pod create fail; alternatively point the controller at an API server that returns 429 for the
first pod create.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artgc-fail-once-
spec:
  entrypoint: main
  artifactGC:
    strategy: OnWorkflowCompletion
  templates:
    - name: main
      container:
        image: alpine:3.20
        command: [sh, -c, "echo hello > /tmp/out.txt"]
      outputs:
        artifacts:
          - name: out
            path: /tmp/out.txt
            s3:
              key: artgc-fail-once/out.txt

After the single failed attempt, remove the deny policy (or let the API server recover) and observe
that the controller never creates the GC pod, the finalizer remains, and kubectl delete wf hangs.

Logs from the workflow controller

{"level":"INFO","msg":"Marking workflow completed", ...}
{"level":"INFO","msg":"adding artifact GC finalizer", ...}
{"level":"INFO","msg":"Creating Artifact GC Task", ...}
{"level":"INFO","msg":"creating pod to delete artifacts", ...}
{"level":"ERROR","msg":"failed to GC artifacts","error":"failed to create pod: Pod \"...-artgc-wfcomp-671565822\" is invalid: metadata.ownerReferences.uid: Invalid value: \"\": must not be empty", ...}
{"level":"INFO","msg":"Workflow update successful", ...}
# then every 20 minutes, forever:
{"level":"INFO","msg":"Processing workflow","Phase":"Succeeded", ...}
{"level":"INFO","msg":"Task-result reconciliation","numObjs":0, ...}

Logs from in your workflow's wait container

Not applicable, the workflow's own pods completed normally.