Skip to content

Cloudflare: admitted durable dispatch never settles and never observes failure; host only recovers via blind timeout (post-#497 nightly) #538

Description

@Divkix

Describe the bug

On Cloudflare Workers, a durable Flue agent submission can be admitted and then leave the host with no terminal outcome:

  • dispatch() / internal Flue admit returns 200
  • the agent DO runs at least once
  • sandbox configure can start
  • the submission never completes its required side-effect tool call
  • observe() never delivers a host-visible failed / aborted settlement

The host only finds out by firing its own blind timeout after the durability window.

This is not a model or docs miss. Later turns in the same thread completed normally in about 7-11s.

Related: #497 ("Cloudflare Sandbox adapter RPC can hang past timeout and block submission settlement").

We are already on a nightly that includes the #497 death-detector fix, so this looks like a remaining silent non-settlement path rather than the pre-fix #497 case.

Environment

  • @flue/runtime / @flue/vite: 2.0.0-nightly.202607240825
  • Cloudflare Workers + Durable Objects
  • Agent durability: { maxAttempts: 3, timeoutMs: 10 * 60 * 1000 }
  • Host reply budget: durability timeout + 60s grace = 11 minutes
  • Worker: cloudflare-community-support
  • Worker version: 86cde636-afc5-4599-ac20-b8976deb17ef
  • Sandbox package: @cloudflare/sandbox@0.12.4
  • Model path: AI Gateway OpenAI Responses (gpt-5.6-luna, thinking max)

Production evidence (one hung turn)

Thread / message id: 1532546670963724439
Dispatch / submission id: sub_01KYTSE0ANS33AYTXHRM048GWG
Window: 2026-07-31T00:33:20Z -> 2026-07-31T00:44:20Z

Time (UTC) Source What happened
00:33:20.041Z Worker Logpush SupportState claims message, Ack posts 200
00:33:21.021Z Worker Logpush Flue internal dispatch POST .../__flue/internal/dispatch 200 (~71ms)
00:33:21.041Z Worker Logpush FlueCfSupportAgent alarm ~35s, outcome=ok, Logs=[], Exceptions=[]
00:33:21.818Z Worker Logpush Sandbox configure succeeds for this turn
00:33:49.806Z Worker Logpush Host claimPost rejected with status=dispatched (no final reply yet)
00:33:55.891Z Worker Logpush Another Flue agent alarm canceled, empty logs
00:33:20 -> 00:44:20 Postgres transcript exactly one dispatch_attempt with dispatch_outcome=accepted; no dispatch_failed
00:33Z AI Gateway analytics gpt-5.6-luna 200s at admit minute, then multi-minute quiet
00:44:20.041Z Worker Logpush + Postgres host-only terminalization: no_reply_timeout / fallback reply, runDurationMs=660001, dispatchAttemptCount=1

What was missing during the hung turn:

  • no post_discord_message
  • no host-visible Flue settlement / failure through our observe() bridge
  • no dispatch_failed transcript row
  • no fiber-interrupt / durability-timeout / submission_settled lines in Worker Logpush
  • the dispatch id shows up in app transcript state, but not in Worker Logpush lines

Control comparison (same thread, minutes later)

Same agent, same Worker version:

  1. 01:03:36Z follow-up answered in ~7s
    • Flue agent logs include post_discord_message + Discord 200
    • sandboxUsed: false
  2. 01:08:22Z another follow-up answered in ~5.5s
    • same pattern

So the runtime/model path was fine. Only the first admitted submission failed to complete or fail in a way the host could see.

Expected behavior

After admission, a durable submission should reach a host-observable terminal state inside the durability window:

  • completed, or
  • failed, or
  • aborted (including durability timeout / abandoned run)

Specifically:

  1. If durability timeout fires, the host should get an observe() failure/abort event with the submissionId.
  2. An agent DO alarm should not be able to end ok with empty logs while the host stays stuck in dispatched until an external blind timeout.
  3. Ideally there is a direct way to read settlement by submissionId after admit (related to Durable submission settlement read/observe by submission id #406 / Expose a terminal result/state observation path for dispatched agent runs #286).

Actual behavior

  • Admission succeeds
  • Some agent/sandbox work starts
  • No required side effect completes
  • No host-visible Flue terminal event arrives
  • Host waits the full durability+grace budget (11 min) and posts its own timeout fallback

From the host's point of view, the submission is neither alive nor dead. It is just silent.

Why this does not look like app misuse

We already use the observe-to-retry pattern from the Flue docs / public API:

  • observe() bridge installed in the agent isolate
  • confirmed failure -> SupportState retry / AI-failure fallback
  • blind timeout only when no confirmed failure arrives

In this incident:

  • retry path never armed (dispatchAttemptCount=1)
  • AI-failure fallback never posted
  • only the blind timeout path fired

That matches "no terminal observe event," not "host ignored a failure."

What we are not claiming

We are not claiming this is proven to be the exact pre-fix #497 sandbox-RPC death hang.

We can prove:

  • silent non-settlement after successful admit
  • no host-visible terminal observe event
  • host-only timeout recovery

We cannot yet prove the exact internal hang site (sandbox RPC vs model/tool loop vs empty fiber exit vs missing settle emission).

Questions

  1. Should durability timeout always emit a host-visible observe() failed/aborted event with submissionId?
  2. In what cases can FlueCfSupportAgent return alarm outcome=ok with empty logs while the admitted submission remains unsettled?
  3. Is there a supported API to poll settlement by submissionId after dispatch() admission?
  4. Any known remaining Cloudflare hang paths after the Cloudflare Sandbox adapter RPC can hang past timeout and block submission settlement #497 death detector?

Notes

We can privately share:

  • sanitized Worker Logpush timeline for the 11-minute window
  • Postgres transcript rows for the hung turn + successful control turns
  • AI Gateway minute aggregates for 00:33Z-01:10Z

Happy to run a minimal reproduction against latest nightly if you have a preferred fixture shape.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions