Replay delivered after a terminal step_failed can exit without any journal write — run stays 'running' forever (observed core 5.0.0-beta.24)

[shtefcs]

Summary

A production run wedged permanently in running after its doStreamStep (from @workflow/ai) failed terminally: the engine did re-deliver the run twice afterwards (both wait_completed events from armed fire-and-forget sleeps), but both replay invocations exited without writing a single journal event — no step_created, no run_failed, no run_completed, no suspension. The run sat running for 15+ hours with zero pending steps and a terminal step_failed as the newest step event.

Filed as documentation of the shape: we observed this on core 5.0.0-beta.24, which is far behind current (beta.43), and we know substantial replay/delivery work has landed since (#3139 → #3198, #2995, #3078). The question is whether this class — replay hydrates a terminal step failure, rejects it into the body, and the invocation then dies silently — is covered by those fixes or is a distinct gap. Happy to re-test on latest and provide anything else; same reporter/app as #3183.

Environment

• @workflow/core@5.0.0-beta.24 (workflowCoreVersion from the run record), workflow@5.0.0-beta.20, @workflow/ai@5.0.0-beta.11 (patched — batching/telemetry hunks; the failing step and the surrounding catch are unmodified paths)
• world: vercel (managed), Next.js app, Node runtime
• runId wrun_01M0J19W6PMED3H23229ZQDE1A, deployment dpl_EGzJqrTTiKAADMA1aipAtMKrXE5V (team automatio-ai) — shared for engine-side lookup

Journal timeline (from workflow inspect events, all times CEST)

11:55:57.535  step_created   doStreamStep (…WQGF)
11:55:57.535  step_started   (attempt 1)
11:56:10-49   step_retrying / step_started  (attempts 2-4)
11:56:59.719  step_failed    — error: 'The prompt is too long: 263607 > 262143' (app-side cause, deterministic)
12:04:26.102  wait_completed (…WQDX — fire-and-forget sleep armed 11:39)
12:14:50.518  wait_completed (…WQFS — fire-and-forget sleep armed 11:49)
— nothing further, ever. Run status 'running' 15h+ until we intervened app-side.

19 of 20 steps completed, 1 failed, 0 pending. Both waits delivered (the events are journaled), so this is not a lost-delivery/dormancy case like #3183 — the invocations ran.

What the app body does at that point

The workflow body awaits the step, catches the hydrated error (it arrives as FatalError: Step "…doStreamStep" failed after 3 retries: …), logs, and then calls another "use step" function as part of its error path. Our function logs show each replay reached the catch (the log line inside it is the invocation's last stdout line) — but no step_created for the follow-up step was ever journaled. So the invocation died somewhere between body-level catch code and the next step registration/suspension commit, twice, identically, with HTTP-level success from the platform's perspective and nothing recorded engine-side.

Reading dist/step.js, the step_failed consumer rejects the hydrated error via ctx.promiseQueue — that part clearly worked (our catch observed the correct error). The silence begins after user code resumes from that rejection.

Why this matters

A run in this state is undetectable and unrecoverable from within the framework: not failed (no run_failed), not completed, no pending primitive for the queue to re-drive beyond already-armed timers — and further deliveries repeat the same silent death deterministically. We shipped an app-side watchdog (external liveness sweep) as containment, but the engine-side invariant "a delivered replay either completes, fails, or suspends the run — always journaling something" seems like the right place for a backstop.

Cross-refs

• beta.37: parallel step batch leaves the run dormant (no error, no queued continuation) when a fire-and-forget sleep() is pending #3183 (ours; different shape — there deliveries never came; here they came and died)
• Step failure publication is lost when a step throws a thenable: normalizeUnknownError awaits the raw value unbounded #2994 (adjacent: step failure publication lost — here publication succeeded, replay-of-failure wedged)