Workflows submitted without the controller instance ID stay Pending forever with no error

[marcosvm]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Filtering by workflows.argoproj.io/controller-instanceid is documented and intentional (Scaling - Instance ID).

The bug is that a mismatch is a successful no-op. Submit returns a created Workflow. The CLI prints Status: Pending. --watch / --log hang until Ctrl-C. There is no Kubernetes event, no controller log line (the informer never sees the object), and no submit-time error. The only CLI output is periodic Re-establishing pod watch, which looks like progress.

Expected: the user is told the Workflow is not owned by any controller (instance ID missing/mismatch, controller down, or namespace not watched).

I could not cluster-test :latest (no cluster). The behavior is present on main (read against bde5adf91). The relevant paths have been this way since #2786.

Easiest way to hit it: CLI without ARGO_SERVER and without --instanceid / ARGO_INSTANCEID. The CLI uses the direct-kube client with an empty instance ID, and instanceid.Service.Label removes controller-instanceid when the client ID is empty. The controller informer selector is controller-instanceid=<id>, so the object is never queued.

The same silent drop happens if argo-server's --instanceid does not match the controller. --instanceid cannot be combined with ARGO_SERVER.

Related: CronWorkflow → Workflow copies the instance ID label; NewWorkflowFromWorkflowTemplate does not. Same shape as #15219, but for the user-submitted Workflow rather than WorkflowTaskSet.

Version(s)

main @ bde5adf91. Not re-run on workflow-controller:latest.

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

# Controller ConfigMap data.instanceID: my-instance
# ARGO_SERVER unset, ARGO_INSTANCEID unset
# argo submit --from workflowtemplate/instanceid-repro -n my-ns --watch
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: instanceid-repro
  labels:
    workflows.argoproj.io/controller-instanceid: my-instance
spec:
  entrypoint: main
  templates:
    - name: main
      container:
        image: argoproj/argosay:v2
        args: [echo, hello]

Logs from the workflow controller

(no output — the controller never sees the Workflow; that absence is the bug)

Logs from in your workflow's wait container

(no pods are created, so there is no wait container)

[Joibel]

This is a "problem" that has existed for many years, and seems to be a one off installation time configuration issue.

I don't see why it's worth adding more code to maintain to fix - persuade me otherwise.

[atirna]

This only applies when an explicit WorkflowTemplate or ClusterWorkflowTemplate already carries a controller instance ID. The submit path checks that existing label before creating a Workflow; unlabeled templates stay unchanged, and it does not try to diagnose controller availability or namespace watches.

It is still configuration, but this is a contradiction the submitter can already see: a successful submit creates a Pending Workflow that no controller can ever observe. I think rejecting that at the existing submit boundary is worth the small check, rather than reporting success with no actionable recovery.

[Joibel]

A server or CLI configured for one instance ID can't list or get templates labelled with a different one, so a user can only hit this by typing the name of a template their client is deliberately configured not to see. In the direct-kube CLI case that's a missing --instanceid, which is a one-off setup mistake rather than something worth a check on every submit. In the argo-server case the workflow doesn't hang at all, it runs under the server's own instance, which is a different question from this issue.

[atirna]

closing this, the direct-kube case is a missing --instanceid setup and the Argo Server path creates under its own instance.