Stateful swarm handoff hangs on main — #1356 stopped registering transfer task defs the SDK still expects

[ling-senpeng13]

Summary

The stateful-swarm agent handoff hangs (workflow stuck RUNNING until swept, ~15 min) on main. Surfaced by the new python SDK e2e gate (PR #1351): e2e/test_suite14_stateful_domain.py::TestSuite14StatefulDomain::test_stateful_swarm_handoff_completes fails with Expected COMPLETED, got RUNNING, reproducibly (2/2 runs, ~908s each).

Regression window / bisect

• 2026-07-17: green (0 failures / 154) against a from-source main server + conductor-ai-e2e-python-2.0.0-rc2 bundle.
• 2026-07-20: hangs, same bundle, only the server advanced.
• The one relevant server change in that window is Enhances A2A/AgentSpan execution #1356 "Enhances A2A/AgentSpan execution" (3ea601884).

Root cause

#1356 stopped registering the swarm transfer / handoff-check task defs — its diff removed the registerTaskDef(...) calls for _check_transfer / _handoff_check and collectDynamicTransferNames(...), with the comment "Generated transfer and handoff-check names are compiler-owned INLINE logic and must never be registered."

But at runtime the pinned SDK bundle (2.0.0-rc2) still registers those transfer task defs via PUT /api/metadata/taskdefs, and the server (post-#1356) no longer creates them, so the update fails:

[http-nio-8080-exec-13] WARN ApplicationExceptionMapper - Error NotFoundException url: '/api/metadata/taskdefs'
com.netflix.conductor.core.exception.NotFoundException: No such task by name swarm_agent_b_transfer_to_swarm_agent_a
	at com.netflix.conductor.service.MetadataServiceImpl.updateTaskDef(MetadataServiceImpl.java:86)

Observed for every swarm transfer name, e.g. swarm_agent_a_transfer_to_swarm_agent_b, e2e_s14_stateful_swarm_transfer_to_swarm_agent_a, swarm_agent_a_transfer_to_e2e_s14_stateful_swarm, and the Suite9 equivalents. With the transfer tasks never usable, the swarm agents can't complete a handoff → the workflow (e2e_s14_stateful_swarm) loops and never terminates.

This is a server/SDK contract break: the server moved transfer/handoff to compiler-owned INLINE, but the SDK bundle it's validated against still expects those task defs registered. #1356's added unit test (SwarmHandoffCompilerContractTest) doesn't cover the registration/execution contract, so the gap only shows end-to-end.

Ask

Reconcile the transfer/handoff contract from #1356 — either restore registration of the dynamic transfer task defs, or ship (and pin the e2e bundle to) the matching SDK that no longer PUTs them. Until then the test is xfail-ed in PR #1351's known-failures-python.json linking here; it will XPASS once fixed.

[kunalrawat425]

I am starting work on this issue.