[feat][agent_rearrange][per-node retry and fallback]

[kyegomez]

Problem

AgentRearrange retries and fallbacks are global — you can't express "retry B three times, and if it still fails, route to C" in the flow itself. Error handling has to live in the caller's try/except, which breaks the declarative model.

Proposed feature

Annotate individual nodes in the flow string with retry and fallback directives:

"A -> B!3 -> C"               # B retries up to 3 times
"A -> B!3>D -> C"             # B retries 3 times, then falls back to D
"A -> B?D -> C"               # B on error routes to D

Design sketch

• Extend the flow parser to recognize !N (retry count) and >X or ?X (fallback agent).
• Each node becomes a small state machine: attempt, on exception retry or fallback or raise.
• Fallback agents are resolved from the same agents list; unknown names fail at parse time.

Files

• swarms/structs/agent_rearrange.py

Why

Puts error handling where it belongs — next to the node that can fail — instead of in every caller.

[adichaudhary]

Working on this

[kinthaiofficial]

Per-node retry and fallback is critical for production swarms. The global-only error handling breaks the declarative model as you noted.

Our circuit breaker pattern for agent nodes:

We use a three-state circuit breaker per node:

• Closed (normal): Requests flow through. Track consecutive failures.
• Open (tripped): All requests immediately return a CircuitOpenError. No API calls made. Timer starts.
• Half-open (testing): After cooldown, allow one request through. If it succeeds, return to closed. If it fails, return to open with longer cooldown.

The circuit breaker is superior to simple retry counts because:

1. It prevents token waste during known-bad periods. A node that failed 3 times in a row is likely to fail again. Retrying immediately wastes tokens. The open state stops all calls until the cooldown expires.

2. Exponential backoff on the cooldown: First trip = 5s cooldown. Second trip = 15s. Third = 45s. This handles both transient errors (short cooldown) and persistent outages (long cooldown).

3. Composable with fallback routing: When the circuit opens, the flow should route to the fallback node. Your proposed syntax (B!3>D) maps naturally to this: retry B 3 times (stays in closed state), if circuit opens, route to D.

Additional design consideration:

Different failure types deserve different handling:

• Rate limit errors: Back off and retry (the request will succeed later)
• Authentication errors: Fail immediately and route to fallback (retry will not help)
• Timeout errors: Retry with longer timeout, then fallback
• Budget errors: Fail immediately, do not retry (budget is exhausted)

The flow parser should support annotating the failure type that triggers the fallback, not just a generic failure count.

Architecture patterns: https://blog.kinthai.ai/221-agents-multi-agent-coordination-lessons