[feature] Add Argo Workflows 4.x compatibility

[jeffspahr]

Feature Area

/area backend
/area testing

What feature would you like to see?

Add support for Argo Workflows 4.x in KFP while preserving compatibility with the current Argo 3.x support window during the migration period.

Today this repo documents support for Argo Workflows 3.5, 3.6, and 3.7, with Argo 3.7 as the default bundled/version-plumbed line. Argo Workflows 4.x is now released upstream, and KFP needs a tracked effort to validate and add 4.x support without regressing existing 3.x users.

Proposed scope:

• add at least one Argo 4.x CI lane while keeping a representative Argo 3.7 lane green
• update vendored Argo manifests, CRDs, RBAC, deploy/version plumbing, and support documentation
• verify compile, upload/run, retry, recurring runs, logs/artifacts, archived workflows, and metadata-writing flows against Argo 4.x
• only introduce narrow compatibility handling where Argo 3.x and 4.x behavior diverges, rather than a broad provider abstraction

What is the use case or pain point?

Users who want to run KFP on clusters with current Argo Workflows releases do not have a supported path today. Supporting only Argo 3.x forces operators either to pin to older Argo versions or to run KFP against an untested Argo 4.x controller.

This is also a good forcing function to identify and harden the most version-sensitive integration seams in KFP, especially around workflow status interpretation, retry behavior, logs/artifacts, metadata labels, and recurring-run interactions.

Is there a workaround currently?

The current workaround is to stay on an Argo 3.7 deployment or to try Argo 4.x without an official compatibility claim. Neither is ideal for operators who need newer Argo fixes and features.

---

Love this idea? Give it a 👍.

[jeffspahr]

/assign

[jeffspahr]

Local spike results from an isolated Argo 4.0.4 probe:

• Added local CI spike lanes for v4.0.4 in .github/workflows/e2e-test.yml and .github/workflows/api-server-tests.yml as explicit include jobs, while leaving the default 3.x matrices unchanged.
• third_party/argo manifest/test/doc update targets rewrite cleanly to v4.0.4.
• kubectl kustomize --load-restrictor LoadRestrictionsNone succeeds for both the vendored Argo base and the standalone KFP overlay with Argo 4 refs.
• In a disposable Kind cluster, the real KFP deploy path (manifests/kustomize/cluster-scoped-resources plus .github/resources/manifests/standalone/default) successfully brought up the Argo 4 workflow-controller in kubeflow.
• A minimal argoproj.io/v1alpha1 Workflow completed successfully under that Argo 4 controller.
• A temporary Go probe using the repo's current github.com/argoproj/argo-workflows/v3 typed client was able to create, get, and observe terminal status for workflows against the Argo 4 controller, including artifactRepositoryRef in status.
• A temporary probe using KFP's current RetrievePodName helper matched the real pod name for a nontrivial step node under Argo 4's workflows.argoproj.io/pod-name-format: v2.

Current read: we should not jump straight to a repo-wide /v3 -> /v4 Go migration. The next step is to run the new Argo 4 CI lanes and use those failures to identify whether the remaining work is in deploy/RBAC, runtime parsing, or specific backend code paths.

[jeffspahr]

CI spike result from PR #13204:

• Added one Argo v4.0.4 standalone API lane and one Argo v4.0.4 E2E critical lane.
• Both new Argo 4 lanes passed:
  • KFP API Server tests Standalone - K8sVersion=v1.34.0 argo_version=v4.0.4 cacheEnabled=true proxyEnabled=false pod_to_pod_tls_enabled=false passed in 15m26s.
  • End to End E2ECritical Tests - K8s v1.34.0 cacheEnabled=false argoVersion=v4.0.4 proxy=false pod_to_pod_tls_enabled=false passed in 24m28s.
• The matching v3.7.3 standalone control lane also passed.
• The matching v3.7.3 E2E critical control lane failed in Run Tests with generic existing-suite churn (workflow.argoproj.io ... not found, missing output files, wait container errors), so this spike did not reveal an Argo-4-specific regression.
• Practical conclusion: Argo v4.0.4 works in the real CI deploy path and the current Argo /v3 Go dependency is sufficient for the tested standalone API and E2E critical flows. This does not justify a broad /v3 -> /v4 migration yet.

PR: #13204